Recommended configuration for cluster setup with DHCP/DNS managed by Incus

Incus
1

Hi all. I’m currently testing an Incus cluster setup in some VMs. My host OS is NixOS. I want to have a cluster setup with a set of shared networks which are not NATed so that guests are accessible directly from outside, while using Incus’s DNS and DHCP feature. I got all of this working on a single node, as well as in a cluster, but in a cluster it seems a little jank, and I wanted to check if I’m missing something. There aren’t many addresses involved here but for clarity I’ll give the addresses from the subnet I’ve been testing with, 10.10.4.0/24.

My Incus config is extremely barebones:

config:
  cluster.https_address: 10.10.0.230:8443
  core.dns_address: 10.10.0.230:53
  core.https_address: 10.10.0.230:8443

First of all, I’m coming from Proxmox, and I have noticed I often try to do things in a Proxmox way before discovering the Incus way of doing things, so apologies if I’ve missed something obvious.

I have VLAN interfaces set up on the host which are unconfigured. In Incus, I have created bridge interfaces for each of these VLAN interfaces, and at first left them unconfigured. This works, instances were able to get a DHCP lease from my DHCP server on my router (10.10.4.1) and were accessible from and had access to my network. Then, I tried getting the Incus DNS integration to work. I turned off my router’s DHCP server on that network, and then re-enabled ipv4.dhcp. Suddenly, the instances wouldn’t get an address. I checked with lsof -n -i :67, and nothing was bound. After searching around and finding nothing, I tried giving the bridge interface an IP address (10.10.4.2), and immediately the DHCP server was started, the instances got addresses, and DNS records were populated and handed off to my DNS server.

This is all well and good, except for when I want to add another node to my cluster. Now, these 2 nodes have the same IP address on their bridge interfaces, which while I can’t think of any specific problems this would cause, seems not ideal. An idea I tried was setting the address of the bridge to the broadcast address of that subnet, 10.10.4.255, which still works, but seems like a very hacky “fix”.

Network config as of now:

config:
  dns.zone.forward: dev.dfsek.com
  ipv4.address: 10.10.4.255/24
  ipv4.dhcp.gateway: 10.10.4.1
  ipv4.dhcp.ranges: 10.10.4.8-10.10.4.99
  ipv4.nat: "false"
  ipv6.address: none
  ipv6.dhcp: "false"
  ipv6.nat: "false"
  security.acls: test,test-in
description: ""
name: br104
type: bridge
used_by:
- /1.0/instances/test
- /1.0/instances/test2
- /1.0/profiles/net_dev
managed: true
status: Created
locations:
- incus-amd64
- incus2-amd64
project: default

So I suppose my question is, is there a correct way to set the interface to “broadcast only” (as that is all that’s needed for DHCP)? Is my method of just setting the address to the broadcast address correct? It works, but my searching has not led me anywhere indicating whether this is correct practice or not. If this is wrong, is there a way to allow each node to have a separate address on the bridge, or should I use a different network setup in my cluster?

1 Like
2

Hello and welcome,

clustering Incus instances doesn’t automatically create a “clustered” network. On default each cluster instance will have it’s own default bridge “incusbr0” configured and manages IP addresses on their own. There are multiple options to setup a clustered network.

  • One is to manage the DHCP and DNS external as you described
  • configure an OVN network
  • use vxlan between your bridges
  • use static addresses very manual approach

Each of them have their pros and cons. Depending on your requirements the first and second are the most straight forward to setup and work with. For a homelab VXLAN would be also a good option.

Incus is very flexible in how it can be configured for your needs but needs certain knowledge and configuration to support your requirements.

3

I’m using NixOS as my host OS for the hypervisors, so I could actually pretty easily give each hypervisor its own static address on the bridge (option 4, if I am understanding correctly), since the specific address wouldn’t matter as it is just for hosting the dnsmasq server. I couldn’t find a way to do that, though, Incus wants every node to have the same address on its corresponding bridge (which is fine as long as NAT is on, but obviously not when NAT is off). That’s why I tried setting the bridge address to the broadcast address, I’m just not sure if that’s good practice or not. I still can’t find anything searching for whether that could cause problems. For reference, here’s what I mean by that:

image

If that is fine to do, it seems like the best compromise for my situation, I did see those other options, but it seems to me like overkill to have encapsulation of the network at that level, when these hosts are all connected to the same L2 network over 40Gbps links. My understanding is that encapsulation with OVN or vxlan would drastically reduce throughput of instances if traffic was flowing between hosts, since that encapsulation is done in software.

4

As mentioned there are pros and cons of each option and you need to wight it out for your use case.

There are a few options how you can tune your network to overcome the overhead added by OVN or VXLAN. On a dedicated network for cluster node communication using jumbo frames is a valid option and the overhead is reduced to nearly nothing compared to the standard MTU.

The simplest approach would is to have a separate DNSMASQ sitting out side of the Incus cluster and provide IP’s and DNS. This way you don’t need to manage IP’s on your own and have full network speed.

5

Is what I am currently doing incorrect? Again, it works, I’m just wondering if it will come back to bite me later, since I can’t find anything suggesting this is a thing people have done before, and it seems a bit sus to me. But I can’t think of any specific reasons why it would stop working, other than it just being something unsupported and weird (is it unsupported or weird? again, no idea)

[dfsek@incus-amd64:~]$ incus network list-leases br104
+----------+-------------------+-------------+---------+--------------+
| HOSTNAME |    MAC ADDRESS    | IP ADDRESS  |  TYPE   |   LOCATION   |
+----------+-------------------+-------------+---------+--------------+
| br104.gw |                   | 10.10.4.255 | GATEWAY |              |
+----------+-------------------+-------------+---------+--------------+
| test     | 10:66:6a:bd:a9:b1 | 10.10.4.89  | DYNAMIC | incus2-amd64 |
+----------+-------------------+-------------+---------+--------------+
| test     | 10:66:6a:bd:a9:b1 | 10.10.4.89  | DYNAMIC | incus-amd64  |
+----------+-------------------+-------------+---------+--------------+
| test2    | 10:66:6a:cc:59:3a | 10.10.4.53  | DYNAMIC | incus2-amd64 |
+----------+-------------------+-------------+---------+--------------+
| test2    | 10:66:6a:cc:59:3a | 10.10.4.53  | DYNAMIC | incus-amd64  |
+----------+-------------------+-------------+---------+--------------+
| test3    | 10:66:6a:22:ff:dc | 10.10.4.39  | DYNAMIC | incus2-amd64 |
+----------+-------------------+-------------+---------+--------------+
| test3    | 10:66:6a:22:ff:dc | 10.10.4.39  | DYNAMIC | incus-amd64  |
+----------+-------------------+-------------+---------+--------------+

[dfsek@incus-amd64:~]$ dig @10.10.0.230 -p 53 axfr dev.dfsek.com

; <<>> DiG 9.20.15 <<>> @10.10.0.230 -p 53 axfr dev.dfsek.com
; (1 server found)
;; global options: +cmd
dev.dfsek.com.		3600	IN	SOA	dev.dfsek.com. 10.10.2.2. 1770873058 120 60 86400 30
dev.dfsek.com.		300	IN	NS	10.10.2.2.
br104.gw.dev.dfsek.com.	300	IN	A	10.10.4.255
test3.dev.dfsek.com.	300	IN	A	10.10.4.39
test2.dev.dfsek.com.	300	IN	A	10.10.4.53
test.dev.dfsek.com.	300	IN	A	10.10.4.89
test3.dev.dfsek.com.	300	IN	A	10.10.4.39
test2.dev.dfsek.com.	300	IN	A	10.10.4.53
test.dev.dfsek.com.	300	IN	A	10.10.4.89
dev.dfsek.com.		3600	IN	SOA	dev.dfsek.com. 10.10.2.2. 1770873058 120 60 86400 30
;; Query time: 37 msec
;; SERVER: 10.10.0.230#53(10.10.0.230) (TCP)
;; WHEN: Wed Feb 11 22:10:58 MST 2026
;; XFR size: 10 records (messages 1, bytes 455)


9: br104: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 10:66:6a:da:1b:2b brd ff:ff:ff:ff:ff:ff
    inet 10.10.4.255/24 brd 10.10.4.255 scope global br104
       valid_lft forever preferred_lft forever
6

I set up a cluster using the example setup, and here the nodes also share the same (automatic) IPV4 address on the bridge. Is it just expected that nodes will have the same addresses on their respective bridge interfaces? Are you not supposed to use the DHCP/DNS functionality at all in a cluster? I’m very confused.

7

Well that depends how you have configured the first initial node. Assume you have used default settings it will configure a local bridge “incusbr0” on each incus cluster node which will have it’s own network settings and DNS etc.

Have a look at this How to make a server a member of more than one VxLAN (ie a member of 2 diff VNID) which should explain it a bit more in detail and how to configure Incus using VXLAN.

One option you have is to give each bridge a different ipv4.dhcp.ranges for example:

  • node1 range 10.10.4.50 - 10.10.4.99
  • node2 range 10.10.4.100 - 10.10.4.149
  • node3 range 10.10.4.150 - 10.10.4.199

As long as the ranges between the nodes are not overlapping you have unique IP addresses.

Alternative disable DHCP on each bridge and use a central DHCP server outside of Incus as mentioned above.

Again Incus provides many different solutions but requires to understand how they need to be configured. Flexibility comes with complexity and clustering Incus is a bit of an advanced setup and needs to be thought through. I would recommend using OVN as setup is simple and solves your issue would also eliminate the “incusbr0” bridge….

8

I’m not sure you’re actually reading my question - or the Incus documentation for that matter. I have explained that I am not using the default configuration, I’ve given my exact configuration in fact.

Also, what you’ve told me to do:

is something the documentation specifically says cannot be done:

The only configuration keys that may differ between networks on different members are bridge.external_interfaces, parent, bgp.ipv4.nexthop and bgp.ipv6.nexthop.

I also have looked into OVN and vxlan and confirmed that for my usecase, it is not what I ought to be using, for many very obvious reasons, mainly that all my nodes are residing on the same L2 network, and never will have to span different networks. The post you linked for vxlan has the user setting up a cluster across separate cloud providers. All my nodes are physical hosts right next to each other. I’m bridging VLAN interfaces, which are already on the same switch. Connecting them isn’t the problem, they are already connected and communicating just fine. Instances across nodes can connect, I can access them from outside the nodes, etc. etc. Unless I am missing something (which again, please tell me what I am missing), each node would still have the same IP address, because it’d still need a bridge, thus bringing me back to square one, but with a massively overcomplicated network setup for my usecase.

To be honest, is quite frustrating to hear you repeatedly say things like this, because it really seems that you yourself do not have this understanding you are saying that I need.

My question remains, is it possible to use the DHCP/DNS functions of Incus on bridges where NAT is disabled? Is my solution of setting the bridge address to the broadcast address a footgun? The only thought I had for how it could be is a potential race condition where 2 instances on different nodes try to get DHCP leases at the same time and the dnsmasq running on each node hands out the same lease, but I have yet to look into specifically how Incus sets up the dnsmasq instances to see whether that would be possible.

It is beginning to seem like the answer to my question is “no” and I should just use an external DHCP server.

9

Sorry for the confusion and you are right in a cluster you can’t define different ranges etc.

No this isn’t possible as far as I know. Incus doesn’t manage IP addresses on it’s own, it uses DNSMASQ under the hood installed on each individual node listening on the managed bridge.

The only option I’m aware about is to disable the DHCP feature for your incus network and use an external service.

10

Yes. There’s nothing magic about NAT: just turn it off. Then add a static route upstream to route your incus subnet via the incus host.

The problem arises if you want to use the same subnet across multiple nodes in your cluster. You can’t have multiple, non-coordinating DHCP servers on the same subnet. dnsmasq isn’t a clustered DHCP server (*).

What I think you can do is run unmanaged bridges on the other nodes, and then trunk them to the managed bridge on the first node. You wouldn’t have any redundancy, since the first node is responsible for DHCP and DNS. Also, all the DNS requests will all go across to the first node. But that may be good enough for what you want.

Otherwise, stick DHCP and DNS in its own container, but you won’t get the incus hostnames added automatically.

(*) EDIT: I don’t use clustering myself, so I’ve not tested this. Possibly, if incus understands the networks are bridged together externally, it will run a single instance of dnsmasq on a node of its choice, and only one node will be active as the gateway. The documentation is unclear here; it says you have to create the same (named) network on all cluster nodes, but not whether they are separate layer 2 domains / subnets or the same.

11

I am aware, thanks. /s

Have I not described my problem well enough? The problem is not that I can’t turn NAT off, that is easy and I’ve described in great detail how I’ve done that. Is there something more you would like to know about my setup? A diagram perhaps? If you want me to provide more details, please ask.

I already have my network set up and working exactly how I want it. The only thing that I do not have working is DHCP/DNS, which it really seems is just unsupported in my network setup.

This is what I had assumed as well, but my testing revealed that every node tries running the DHCP server and assigns itself an address at once (once again, something I have described in great detail above). My goal with asking this question was to find out if it is possible to get it working as you have described here, which it seems it is not.

My guess is that Incus wants every host’s network to be a separate L2 network, although this is never specifically stated anywhere. At this point I’m just going to go look at the code myself.

12

I believe I have answered my question. I have more questions than answers now. Looking at the code, if I am understanding correctly Incus does not use dnsmasq to generate the lease, even if the instance has a dynamic IP. Incus chooses the address and hands off the reservations for dnsmasq to serve.

Here is the function to allocate the address: incus/internal/server/dnsmasq/dhcpalloc/dhcpalloc.go at main · lxc/incus · GitHub

Here is the code in the bridge interface to rebuild the leases file when an instance is started up: incus/internal/server/device/nic_bridged.go at main · lxc/incus · GitHub

I followed the logic and was pretty sure this was how it worked, but to sanity-check I grabbed the commandline of the dnsmasq that Incus was running and accessed the leasefile it generated:

[dfsek@incus-amd64:~]$ cat /var/lib/incus/networks/br104/dnsmasq.leases
1771172144 10:66:6a:22:ff:dc 10.10.4.39 test3 01:10:66:6a:22:ff:dc
1771172143 10:66:6a:cc:59:3a 10.10.4.53 test2 01:10:66:6a:cc:59:3a
1771172143 10:66:6a:bd:a9:b1 10.10.4.89 test 01:10:66:6a:bd:a9:b1

Sure enough, here they are. Unsure though whether Incus is putting those there or if dnsmasq is. For now let’s assume that it’s Incus.

Now that we can assume that instances fighting over IP addresses won’t be an issue, on to the problem of each node having the same IP on the bridge. This can simply be solved with firewall. We don’t care which device has the DHCP server’s IP, so long as there are no points in the network at which multiple IPs will receive or respond to a packet. We can do this simply by not allowing any traffic in or out of the bridge to or from the bridge’s IP address. That way each host’s DHCP server serves only instances on that host and nothing more. I used the following nftables ruleset to do this:

table bridge filter {
	chain output {
		type filter hook output priority 0; policy accept;
		oifname "vlan104" drop
	}

	chain input {
		type filter hook input priority 0; policy accept;
		iifname "vlan104" drop
	}
}

This forbids any traffic not originating on the host (replace vlan104 with your external bridged interfaces) from entering or leaving the host’s bridge. Traffic between instances on different hosts is allowed.

Some sanity checks:

  1. Each instance gets only one lease:

    [dfsek@incus-amd64:~]$ incus network list-leases br104
+----------+-------------------+-------------+---------+--------------+
| HOSTNAME |    MAC ADDRESS    | IP ADDRESS  |  TYPE   |   LOCATION   |
+----------+-------------------+-------------+---------+--------------+
| br104.gw |                   | 10.10.4.254 | GATEWAY |              |
+----------+-------------------+-------------+---------+--------------+
| test     | 10:66:6a:bd:a9:b1 | 10.10.4.89  | DYNAMIC | incus-amd64  |
+----------+-------------------+-------------+---------+--------------+
| test2    | 10:66:6a:cc:59:3a | 10.10.4.53  | DYNAMIC | incus-amd64  |
+----------+-------------------+-------------+---------+--------------+
| test3    | 10:66:6a:22:ff:dc | 10.10.4.39  | DYNAMIC | incus2-amd64 |
+----------+-------------------+-------------+---------+--------------+

  2. Gateway IP is not accessible from outside. (Ping the gateway IP from any other host)

  3. Instances can communicate with each other

    root@test7:~# ping 10.10.4.15
PING 10.10.4.15 (10.10.4.15) 56(84) bytes of data.
64 bytes from 10.10.4.15: icmp_seq=1 ttl=64 time=1.43 ms
64 bytes from 10.10.4.15: icmp_seq=2 ttl=64 time=0.663 ms
64 bytes from 10.10.4.15: icmp_seq=3 ttl=64 time=0.671 ms

    test and test7 reside on different nodes.

  4. No two instances get the same IP address:
    I restricted the range by setting ipv4.dhcp.ranges to only 10 addresses, 10.10.4.8-10.10.4.18. I then created 10 instances one by one, alternating which host they were created on. This test proved… problematic. Each time I repeated, 2 instances (the same 2, interestingly, even after clearing the lease file manually) did get the same address, and checking the leases file, some instances’ leases got shared between hosts, but some did not, which seems to be causing the problem. Is this a bug? I did not see any logic in the DHCP code regarding cluster members, though I’m also not sure what to look for/at what level clustering information is distributed.

Please let me know if I’ve missed anything, or am misinterpreting the linked code. This seems like a bug since it would appear that logic to distribute DHCP reservations exists, yet 2 instances can still occasionally get the same address. I think I’ll stick with my external DHCP server for now. I would like to get Incus’ DHCP/DNS working, though. The premise seems sound, distribute an Incus-generated reservation to all the running dnsmasq instances in the cluster before the instance is started. It just does not seem to be functioning as intended(?)

Please let me know if I should open an issue about this, or if what I’m doing is an unsupported usecase.

13

I did find a post from… 2018, saying that DNS/DHCP is meant to be externally managed in clusters. surely this has changed in the 8 years since.

LXD clustering right now very much assumes that you have an existing layer-2 infrastructure and that you’ll effectively just plug your containers into an existing physical network (VLAN) with a router provided DHCP/DNS for you. That’s the case which works best and if integrated with MAAS, LXD can even do IP management for you in this case.

Emphasis mine, since I am aware of all the other things in that comment already and I’m sure that it’ll be pointed out to me yet again if I don’t say so. (once again, as described, I have this setup. The exact setup being described here, the one with VLANs, is my setup, which I have and it works. I am specifically asking about DHCP/DNS)

Is this still the case? Are you assumed to use external DNS/DHCP in an Incus cluster? If so, what is all the logic around DHCP leases for?

14

Since you have to create the networks separately on each cluster node, can you create a different IP address for each node?

# on node 1
config:
  ipv4.address: 192.0.2.1/24
  ipv4.nat: "false"

# on node 2
config:
  ipv4.address: 192.0.2.2/24
  ipv4.nat: "false"

# on node 3
config:
  ipv4.address: 192.0.2.3/24
  ipv4.nat: "false"

As you say, the DHCP servers would race with each other, so the container will effectively get a randomly-picked gateway and DNS server.

Are you assumed to use external DNS/DHCP in an Incus cluster? If so, what is all the logic around DHCP leases for?

Presumably for the non-clustered case, when it works fine.

There are good reasons why I don’t use incus clustering and I think you may have provided an extra one.

15

No. Even if this was possible, it still would not solve the problem, and I have described in excruciating detail why.

16

After so much effort, you still haven’t achieved what you want. May I ask why you have to use Incus’s DNS and DHCP feature for cluster? An external DHCP server with some unmanaged bridges will fulfill what you want.

Even though I’m not using cluster. My instances attached to unmanaged bridge get ip addresses from router. I don’t see why instances in cluster won’t get ip using DHCP.

17

I obviously don’t “have to”. It’s simply a neat feature and I’d like to get it working. I’m willing to fix and PR it myself if this is in fact a bug, even. And being honest, most of my effort here has simply been explaining the same things over and over again, and looking into things I’d already ruled out to see if I’d missed anything, after they were repeatedly suggested to me in the vaguest way without actually explaining how they would solve this problem.

If anyone on the Incus team can comment as to whether this DHCP behaviour is intentional or a bug, I could possibly begin working on making it work in a cluster. I just don’t want to start working on it if it’s already working as intended and you simply aren’t supposed to use these features in a cluster.

18

But incus cluster doesn’t create virtual networks between nodes.

cluster
cluster1210×587 378 KB

Even you find a way, incus managed bridges will never bind eth0, so there will never be a layer 2 switch between nodes’ ibr0, so arp will never broadcast between one node’s ibr0 to the other, so overlapping ips will likely to happen.

flow
flow1092×340 77.2 KB

And you can’t control incus’s dnsmasq, so each node will have a working DHCP server, so each node’s instances will get ip from the node’s dnsmasq, so overlapping ips will likely to happen.

What will happen if you create a managed bridge without ipv4? Instances connected to this bridge won’t have ipv4 network connection.

19

It is clear to me that you have not read anything in this thread.