Instance Networking across cluster members

[MarkDavid]

Good day!

I have recently picked up Incus as a technology I want to use for my Homelab needs. As a context - I do not have a lot of experience with networking, still on a path of learning, as such the following questions might not be formulated too well.

I wanted to see whether it was possible for Container Instances deployed in different Cluster Members to communicate to one another if they are part of the same configured network.

In this case, to test it, I have created two VMs locally, installed Incus and set up a cluster.

root@debian12:/home/vagrant# incus cluster list -c nurs
+------+-----------------------------+------------------+--------+
| NAME |             URL             |      ROLES       | STATUS |
+------+-----------------------------+------------------+--------+
| vm1  | https://192.168.121.38:8443 | database-leader  | ONLINE |
|      |                             | database         |        |
+------+-----------------------------+------------------+--------+
| vm2  | https://192.168.121.53:8443 | database-standby | ONLINE |
+------+-----------------------------+------------------+--------+

root@debian12:/home/vagrant# incus network list -c ntm4us
+------------+----------+---------+---------------+---------+---------+
|    NAME    |   TYPE   | MANAGED |     IPV4      | USED BY |  STATE  |
+------------+----------+---------+---------------+---------+---------+
| eth0       | physical | false   |               | 0       |         |
+------------+----------+---------+---------------+---------+---------+
| lo         | loopback | false   |               | 0       |         |
+------------+----------+---------+---------------+---------+---------+
| my-network | bridge   | true    | 10.80.34.1/24 | 3       | CREATED |
+------------+----------+---------+---------------+---------+---------+

root@debian12:/home/vagrant# incus list -c ns4L
+-------------------+---------+---------------------+----------+
|       NAME        |  STATE  |        IPV4         | LOCATION |
+-------------------+---------+---------------------+----------+
| ubuntu-container  | RUNNING | 10.80.34.247 (eth0) | vm1      |
+-------------------+---------+---------------------+----------+
| ubuntu-container2 | RUNNING | 10.80.34.173 (eth0) | vm2      |
+-------------------+---------+---------------------+----------+
| ubuntu-container3 | RUNNING | 10.80.34.195 (eth0) | vm1      |
+-------------------+---------+---------------------+----------+

Basically, all the containers are attached to the my-network network for the cluster.
The problem is ubuntu-container and ubuntu-container3 can communicate with one another, but cannot reach the ubuntu-container2, and vice versa.

I feel like having communication like this is not possible to be done that easily, but I would expect that this would be a possibility.

For the situation described above, I have several questions:

1. Is the above even possible?
2. If it is possible, is it recommended?
3. If it is possible - how would I go about doing so? I do not need detailed instructions, if possible, point me to the technologies and methods I could potentially use, so that I could learn them myself.

Thank you kindly.

[simos (Simos Xenitellis)]

Hi!

I suppose you tried the steps from How to configure networks for a cluster - Incus documentation

Specifically, you used the following to create the networking.

incus network create --target server1 my-network
incus network create --target server2 my-network
incus network create --target server3 my-network
incus network create my-network

update
It did not work for me either. Something is missing.

1. I have used 3 VMs with Debian Trixie, because it has Incus Stable 6.0.1 in the repositories.
2. I created the cluster network interface, clusterbr0, per instructions.
3. One VM would not receive the MAC address of the other VM for the clusterbr0 network interface.
4. I then populated the ARP table manually with the appropriate MAC address.
5. The ICMP packet would make it to the other VM, but the destination VM would not respond.
6. resolvectl is not getting autoconfigured.

Link 3 (clusterbr0)
    Current Scopes: LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

[simos (Simos Xenitellis)]

@stgraber

I have tried a few times with Incus (6.0.1) to create a cluster with three members in Incus VMs, create networking and get the containers in the cluster to access each other. I could not get the containers in the cluster to ping each other, if they are on different nodes of the cluster.

I created the network my-network using these instructions, How to configure networks for a cluster - Incus documentation

incus network create --target server1 my-network
incus network create --target server2 my-network
incus network create --target server3 my-network
incus network create my-network

I then added the my-network into the default profile and started creating containers.
When I tried to ping from within a VM the containers that resided on other nodes of the cluster, the packets were not routed to the appropriate destinations.

Is it possible to get an Incus cluster to work, and containers access each other through the network, without resorting to more complicated network solutions?

[victoitor (Victor)]

Since no type is provided, the network you’re creating should be of type bridge. Incus creates a bridge in each machine, which is local unless you set it up properly.

Add an interface on each machine which is unconfigured and then see the part which mentions

The only configuration keys that may differ between networks on different members are bridge.external_interfaces, parent, bgp.ipv4.nexthop and bgp.ipv6.nexthop.

So on the creation part of each machine, add bridge.external_interfaces to the unmanaged interface. If those interfaces are connected to a common switch which has a dhcp server running, make sure to configure the created network not to have dhcp.

Don’t have the time to test it right now, but this should work.

Second solution

Also note that if this is the only external interface, then there might be a problem of machine connectivity. If there is precisely one interface on your machines (which is my case), I do basically the same thing but I do this on the host OS of each machine. Then I just use an external bridge on each machine and they all connect to the same network and can ping each other.

On this solution, your default profile network device looks like this:

eth0:
  name: eth0
  nictype: bridged
  parent: br0
  type: nic

Also works just fine for a cluster and they all can ping each other.

[MarkDavid]

Thank you both for the assistance here.

As I am testing all of this within VMs, it should not be a problem introducing a virtual switch, introducing additional interfaces and connecting the VMs to it.

I should get some free time later this week and see if I can implement both of these solutions and I will report back with the results.

[simos (Simos Xenitellis)]

Thanks for this. The gist I get is that I cannot use some managed network created by Incus, in order to provide network connectivity between the containers of a cluster. Such a setup works beautifully with containers but not on clusters.

The easiest way to add networking to a cluster is that one I show above (also found in the Incus documentation). As you mention, it creates local bridges.
In a way it’s counter-intuitive because when I run incus network list,
it shows a single bridge that is used by all nodes of the cluster. It assumes that containers created in a cluster can communicate with each other over the cluster network.

ubuntu@node1:~$ incus network list
+------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
|    NAME    |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| clusterbr0 | bridge   | YES     | 10.173.183.1/24 | fd42:bca1:5c97:11bc::1/64 |             | 3       | CREATED |
+------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| enp3s5     | physical | NO      |                 |                           |             | 0       |         |
+------------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
$ 

In another way, such a default network setup for clusters isolates (for networking) the containers per node. The containers of a specific node can talk to each other, and of course each access the Internet.

If I stop a container, then incus move mycontainer1 --target node2, then start again, the container has joined the local bridge of the new node of the cluster.

When I ping to 10.173.183.1 (in my case, the IP address of the local bridge), that IP address is just the IP address of a local bridge. It’s just the same IP address for each of those different local bridges (one per node of a cluster).

When trying to explain how a cluster works, I think it’s important to show this minimal but not very useful setup. Sure, the containers of one node cannot communicate with those of another node.
Perhaps each cluster node can be assumed to be a business client, hence the network separation among clients is desirable. But if one client sells a server to another client, I can incus move in the easiest and most comfortable way.

In my setup I have three Incus VMs (node1, node2, node3), that each has a enp3s5 network interface (emulated through Qemu?).
I tried several combinations,

1. creating a bridge between enp3s5 (in the VM) with the Incus clusterbr0 I saw above. It messes up some things and I think it is not the right way? (brctl addif).
2. using brctl to create a bridge on enp3s5 called clusterbr0 (brctl addbr) on each VM, then use clusterbr0 as the network interface for the containers. Strangely, it worked for two out of the three VMs in the cluster, but not on the third.

I suppose I create a bridge on the host, then pass it to each of the VMs. Is that right? This give me the impression that I am exposing the containers of the cluster to the host.
How would I pass a bridge from the host to the VMs?

[xarufagem (Gigadjo)]

Hello,

I think you shouldn’t think about networking as a “whole” :

• You need connectivity between hosts running incus (let’s call it ‘management network’, where all incus internal traffic is happening, moving instances from hosts to hosts in a cluster, database replication…)

• You need connectivity for your virtual-instances (let’s call it ‘virtualization network’), it is used for VM/CT to communicate with a network, be it the one the ‘Host’ lies in or, another

In some setup, having hosts & VM/CT in the same network is not a big issue (typically Homelab), but in most of the case, you don’t want your VM to be able to talk to the virtualization host they run on !

Easy setup : use Mac-Vlan to connect your VM/CT to your single network port, a basic limitation of MACVLAN is that, host cannot talk to instances

Less Easy setup : set up a vlan on each of your host (refer to your network config tool, on how to do it…) then set it as primary interface for a bridge on each host (also see your network conf tool), then connect your instances to it as victoitor suggested :

eth0:
  name: eth0
  nictype: bridged
  parent: br0
  type: nic

(adjust to your needs)

You must provide DHCP on this vlan. All instances across all hosts connected to that VLAN will have access to all instances on all hosts (very basic setup).

It is how it’s ‘generally’ done, even if all sorts of technology can be used inbetween, one can for example setup vxlan / gre and plug such an interface in the bridges plugged to virtual instances instead of VLAN, to allow for L2 DHCP / RA to go through…

If ‘external networking’ is not an option, one could pay a look to OVN, as it allows for “automatically” doing many things, within an host, but shared networks, still require L2 to my actual knowledge of OVN/OVS, be it plain Ethernet or an encapsulated form

The problem your report @MarkDavid comes from how incus creates internal bridges : those are not ‘directly’ linked to your interface, instead, some IPTABLES rules are put in place to allow for dnsmasq to provide dhcp/dns, not poisoning your existing network with DHCP Offers / RA.

Instanes can communicate if they are located on the same host (container & container3 are on the same ‘virtualization host’ vm1, not container2…).

If having your instances all connected to the same network as your hosts is what you want, simply create a bridge, on every virtualization host, containing the main interface of your virtualization host (VM1 / VM2). Then plug your instances to it.

I bieleve it can be a little bit confusing
Hope it helps

[victoitor (Victor)]

@simos

The gist I get is that I cannot use some managed network created by Incus, in order to provide network connectivity between the containers of a cluster. Such a setup works beautifully with containers but not on clusters.

You can. Just create a bridge and connect your ethernet interface into it. It can be done through incus just fine. I have done this before. There is just an issue you might have if this is your only nic, as your computer will only have connectivity after incus starts. You have to make sure that your computer does not wait for that interface to come online while you’re booting. Because of this possible issue, I prefer to set the same thing, which is possible through incus, through the OS itself.

[xarufagem (Gigadjo)]

I bieleve the problem, is making a difference between ‘Incus Managed bridges’ and ‘Bridge’

For example, a working bridge conf would look like this (netplan)

    bridges:
        br_lan:
            dhcp4: true
            interfaces:
                - eno1
        br_vlan5:
            dhcp4: no
            interfaces:
                - vlan5

In incus, instance config :

  eth0:
    nictype: bridged
    parent: br_lan
    type: nic

To have it connected to a vlan, i have previously configured on an interface, i use br_vlanX
Or to connect directly to host network, ‘br_lan’

Vlan definition is pretty straightforward (netplan) :

        vlan5:
            id: 5
            link: eno1

Refer to netplan.io, if your distro is using it, like wine, it’s becoming beter with time

On the other hand, Incus Managed Bridges, while still using linux bridge, are ‘auto-configured’ in a different way

Incus Managed bridges provides dhcp / dns by running Dnsmasq, which is cool, but must not be allowed to go through your network card, else it would poison your existing network (If you’re using dhcp on your network for example… it would probably make it a mess… )

[victoitor (Victor)]

Let me just give an overview of how I add instance networking between members of a cluster. In my case, I’m using Debian 12 which is using network configuration defined in /etc/network.

First, I remove the default configuration for each network interface provided in the installation as follows.

/etc/network$ cat interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug enp3s0
iface enp3s0 inet manual

Then I add a host bridge as follows.

/etc/network$ cat interfaces.d/br0 
## dhcp ip config file for br0
auto br0
iface br0 inet dhcp 
	bridge_ports    enp3s0
	bridge_stp	off			# disable Spanning Tree Protocol
	bridge_waitport	0			# no delay before a port becomes available
	bridge_fd	0			# no forwarding delay

This corresponds to a setup in which the IP from the cluster machine is actually obtained from the bridge. The network interface is connected to this bridge.

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 74:56:3c:6a:02:ee brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:74:c0:ea:84:20 brd ff:ff:ff:ff:ff:ff
    inet 10.11.16.11/21 brd 10.11.23.255 scope global dynamic br0
       valid_lft 4987sec preferred_lft 4987sec
    inet6 fe80::6074:c0ff:feea:8420/64 scope link 
       valid_lft forever preferred_lft forever

In this case, each machine has a bridge which is connected to a central switch, so all containers can communicate through the central switch.

Can this be managed by incus? It can! The connection of the local machine nic can be made through the bridge.external_interfaces. But there are just some things which you must take care. As I mentioned before, you have to consider the internet connectivity of your machine itself, but it’s not an issue if you configure it properly. But there is also another small issue. Incus provides a dhcp server for every local bridge interface it manages. On this setup, with all of these bridges connected, you will have dhcp conflicts, since there should be only one running.

To solve this dhcp issue, the easiest sollution is to turn off dhcp in all local bridges and have someone else be the dhcp server (incus can do this with an OpenWRT container, for example). I haven’t checked if this can be turned off on each machine of a cluster if it’s managed through incus.

[simos (Simos Xenitellis)]

Thanks @xarufagem and @victoitor for this.

I could not find /etc/network/ in Debian 12 (images:debian/12/cloud VM). I could find some systemd network configuration but no leads on how to set that up.
I’ll stick to Ubuntu 24.04 at the moment, that uses netplan and accepts netplan apply. There are examples online on how to setup bridges with netplan.

First, I created my three VMs, which are the nodes of the prospective cluster.

incus launch images:ubuntu/24.04 --vm node1
incus launch images:ubuntu/24.04 --vm node2
incus launch images:ubuntu/24.04 --vm node3

This is the default network configuration in the images:ubuntu/24.04 image.

ubuntu@node1:~$ sudo cat /etc/netplan/10-lxc.yaml 
network:
  version: 2
  ethernets:
    enp5s0:
      dhcp4: true
      dhcp-identifier: mac
ubuntu@node1:~$ 

I changed the network configuration to the following a la netplan. The output of ip a looks the same as your output.

ubuntu@node1:~$ sudo cat /etc/netplan/10-lxc.yaml 
network:
  version: 2
  renderer: networkd

  ethernets:
    enp5s0:
      dhcp4: false 
      dhcp6: false 

  bridges:
    br0:
      interfaces: [enp5s0]
      parameters:
        stp: true
        forward-delay: 0
      dhcp4: true
      dhcp6: true
ubuntu@node1:~$ sudo netplan apply
ubuntu@node1:~$ 

I then installed incus and initialized it, specifying that the cluster should use the network interface br0.

ubuntu@node1:~$ sudo incus admin init
Would you like to use clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.10.10.102]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=node1]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: 
Do you want to configure a new remote storage pool? (yes/no) [default=no]: 
Would you like to use an existing bridge or host interface? (yes/no) [default=no]: yes 
Name of the existing bridge or host interface: br0
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]: yes
config:
  core.https_address: 10.10.10.102:8443
networks: []
storage_pools:
- config: {}
  description: ""
  name: local
  driver: dir
profiles:
- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      nictype: bridged
      parent: br0
      type: nic
    root:
      path: /
      pool: local
      type: disk
  name: default
projects: []
cluster:
  server_name: node1
  enabled: true
  member_config: []
  cluster_address: ""
  cluster_certificate: ""
  server_address: ""
  cluster_token: ""
  cluster_certificate_path: ""

ubuntu@node1:~$ 

Then, repeat accordingly for node2 and node3 (i.e. add the netplan configuration, then sudo netplan apply. Finally, sudo incus admin init with the appropriate configuration to setup the cluster.

Subsequently, create a container for each node.

ubuntu@node1:~$ incus launch images:alpine/edge/cloud mycontainer1 --target node1
Launching mycontainer1
ubuntu@node1:~$ incus launch images:alpine/edge/cloud mycontainer2 --target node2
Launching mycontainer2
ubuntu@node1:~$ incus launch images:alpine/edge/cloud mycontainer3 --target node3
Launching mycontainer3
ubuntu@node1:~$

Finally, test whether I can ping from one instance (on a specific node) to another instance (on another node).

ubuntu@node1:~$ sudo incus shell mycontainer1
mycontainer1:~# ping -c 3 mycontainer2
PING mycontainer2 (10.10.10.113): 56 data bytes
64 bytes from 10.10.10.113: seq=0 ttl=64 time=0.293 ms
64 bytes from 10.10.10.113: seq=1 ttl=64 time=0.228 ms
64 bytes from 10.10.10.113: seq=2 ttl=64 time=0.235 ms

--- mycontainer2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.228/0.252/0.293 ms
mycontainer1:~# ping -c 3 mycontainer3
PING mycontainer3 (10.10.10.135): 56 data bytes
64 bytes from 10.10.10.135: seq=0 ttl=64 time=0.278 ms
64 bytes from 10.10.10.135: seq=1 ttl=64 time=0.232 ms
64 bytes from 10.10.10.135: seq=2 ttl=64 time=0.227 ms

--- mycontainer3 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.227/0.245/0.278 ms
mycontainer1:~# ping -c 3 mycontainer2.incus
PING mycontainer2.incus (10.10.10.113): 56 data bytes
64 bytes from 10.10.10.113: seq=0 ttl=64 time=0.236 ms
64 bytes from 10.10.10.113: seq=1 ttl=64 time=0.225 ms
64 bytes from 10.10.10.113: seq=2 ttl=64 time=0.212 ms

--- mycontainer2.incus ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.212/0.224/0.236 ms
mycontainer1:~# 

The downside of all this, is of course that those instances in the cluster can access other instances on the Incus installation on the host. Is it possible to separate those cluster instances from the instances on the host? The cluster instances are in VMs and does not feel natural to access the other instances on the host.

It’s not clear how to setup the DHCP server. With what you are suggesting, are those cluster instances going to be separate from those of the host?

[victoitor (Victor)]

In this solution, they are in the same network as the hosts. For me, that’s fine as I just want a simple solution.

If you want to separate them, then you need to use VLANS which involves setting this up in your switch (which has to be managed) and possibly in your router. But if the solution you’re looking for starts to become too complex, then you might as well just run OVN as it’ll be easier to manage from inside Incus.

At my place, I have a router server running Incus and which runs OpenWRT in a container which is my main router. It also has proper network segmentation through VLANs.

Well, anyway, I don’t recommend doing complex network configurations if you don’t feel comfortable. This is unless you want to study networking, which was my case a while back (I don’t even work with this, it was just curiosity).