How to make a server a member of more than one VxLAN (ie a member of 2 diff VNID)

bmullan · July 8, 2025, 7:47pm

This general config works for 3 test nodes (node1, node2, node3).

Each node is on one of 3 diff Cloud (AWS, Digital Ocean, Hetzner)

On each node I create an incus bridge named brvxlan with a VNID=10
I can then create a container cnx on each Node using the network brvxlan

From any node I can then ping the cnx container on any other node successfully.

I know VxLAN supports a server configured to belong to more than 1 VNID.

My testing goal was to change:

Node1 = VNID 10
Node2 = VNID 10, 100
Node3 = VNID 100

so

Node1 & Node2 are on VNID 10 and their containers can talk to each other

Node1 is not on VNID 100 so it can not talk to Node3 (VNID100)

but
Node2 is also a member of VNID 100 so it can talk to Node3 (VNID 100)

The above should:
let Containers on Node1 talk to Containers on Node2
let Containers on Node2 talk to Containers on Node3

On each Node I create a VXLAN bridge named brvxlan using this general command:

incus network create brvxlan
tunnel.server1.protocol=vxlan
tunnel.server1.id=10
tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2
tunnel.server1.remote=10.0.1.3
ipv4.address=none
ipv6.address=none

can an Incus vm/container be created and attached to 2 different vxlan bridges?

brvxlan10
brvxlan100

or is there a way to specify 1 bridge is a member of 2 VNID? (preferred option if it exists)

I was hoping Incus supported something like 1 of these 2…

Note:
I tried entering both of these and using following syntax, neither worked but if the capability does exist then I would like to know how to configure them both for 1 bridge?

incus network create brvxlan
tunnel.server1.protocol=vxlan
tunnel.server1.id=10
tunnel.server1.id=100
tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2
tunnel.server1.remote=10.0.1.3
ipv4.address=none
ipv6.address=none

OR

incus network create brvxlan
tunnel.server1.protocol=vxlan
tunnel.server1.id=10, 100
tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2
tunnel.server1.remote=10.0.1.3
ipv4.address=none
ipv6.address=none

I’ve been searching all afternoon but no luck finding anything.

Any comments, suggestions ?

candlerb · July 9, 2025, 9:19am

Yes. Simply create two incus networks, and create the VM/container with two NICs (e.g. eth0 and eth1) connected to the two networks. Or make a profile which has both eth0 and eth1.

The question isn’t clear. Are you trying to make a single broadcast domain which covers all three nodes, by bridging/meshing the nodes together? i.e. each node has an IP address on the same subnet, and all three can see each other?

Or, do you want one broadcast domain connecting nodes 1-2, and a separate broadcast domain connecting nodes 2-3 ? Then there would be two different subnets/vxlans, and node 2 would have to do routing if you wanted traffic to flow from node 1 to node 3.

bmullan · July 9, 2025, 2:29pm

I don’t know if the following explanation will help or not explaining my questions…

In VxLAN a Node only responds to traffic on a VNI it is a member of.

In a DC a Server/Node can be configured with more than one VNI.
Traffic on one VNI is not visible to Traffic on a different VNI

In my orig post I mentioned ‘Nodes’:

Node1 = VNID 10
Node2 = VNID 10, 100
Node3 = VNID 100

Just because Node2 is a member of both VNI 10 & VNI 100 does not mean its acting like a bridge for traffic between Node0 and Node3 as those 2 Nodes are not members of the same VNI. In concept kind similar to using IP:Port.

There are many use-cases where a Server/Node should belong to more than one VNI (VxLAN Network Identifier).

VxLAN & Network Segmentation:
VxLAN is a network virtualization technology that uses VNIs to segment physical networks into multiple virtual, Layer 2 networks. This is particularly useful in multi-tenant environments like data centers (DC), where you need to isolate traffic from different users and/or applications.

A Server in Multiple Segments:
A single Server/Node might host multiple VMs/Containers w applications.
Each VM or application instance might need to belong to a different network segment for security, policy enforcement, or other reasons.

VTEPs and VNI-VLAN Mapping:
VxLAN relies on VxLAN Tunnel Endpoints (VTEPs)*, which can be hardware switches or software running on hypervisors, to encapsulate and decapsulate Ethernet frames within UDP packets. VTEPs also manage the mapping between VNIs and local VLANs.

How it Works:

A server/Node connected to a VTEP can have traffic associated with different VLANs.

A VTEP maps each of these VLANs to a unique VNI.

When the server sends traffic, the VTEP encapsulates the Ethernet frame into a VxLAN packet with the corresponding VNI.

This encapsulated packet is then sent across the underlying network.

At the receiving end, another VTEP decapsulates the packet and delivers the original frame
to the destination ‘device’ within the correct/same VLAN and VNI segment.

In essence, a Server/Node (ie my Node2) can logically be part of multiple VxLAN networks by having its interfaces associated w different VLANs that are then mapped to unique VNIs by the VTEP it’s connected to…

This allows for flexible and scalable network designs, especially in large mesh network and/or cloud environment.

All this is useful in a multi-tenant environment

If you “could” configure on Node2 an Incus VxLAN bridge with both VNI 10 and VNI 100.
Then VM/Container services attached to that Node2 Incus bridge are available to any Incus VM/Containers attached to an Incus VxLAN bridge on Node1 or Node3 that has a matching VNI.

“tenant” traffic isolation to/from an any Node’s Incus Bridge even over the intranet or Internet.

“tenant specific services” are isolated to tenants that are members of the correct VxLAN VNI

How this relates to my orig post

Using Incus, to resolve the problem for Node2… I can think of only 2 options to implement a “Node2” connected to multiple VNI.

if an Incus VxLAN bridge does or can support being configured w more than 1 VNI
if an incus VxLAN bridge can not have more than 1 VNI then you could create 1 VxLAN bridge for each VNI (re brvni10, brvni100) but this presents another problem:

some services implemented in VM/Containers may be useful to users/members of VNI 10 but also to users/members of VNI100. But if a users is only a member of one of those VNI but the VM/Container is attached to an Incus VxLAN bridge on the other VNI its not available to those users.

It just seems problematic to me to have to create a identical VM/Container behind an Incus VNI 10 bridge and an identical VM/Container behind an Incus VNI 100 bridge (the 1 sources of truth problem).

That’s why I asked if a VM/Container can somehow be attached to 2 diff Incus VxLAN bridges… I guess kind of like a “symlink” where a VM/Container lives attached to 1 VxLAN bridge/VNI but "virtually appears to be attached to a 2nd VxLAN bridge/VNI also so a VNI 10 user/tenant can access it but so can a VNI 100 user/tenant w/out VM/Container duplication.

candlerb · July 9, 2025, 3:12pm

Yes.

However, you also wrote this:

incus network create brvxlan
tunnel.server1.protocol=vxlan
tunnel.server1.id=10
tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2
tunnel.server1.remote=10.0.1.3
ipv4.address=none
ipv6.address=none

There, it looks like you want vni 10 to connect to both 10.0.1.2 and 10.0.1.3 (although it’s unclear to me if you’re allowed to repeat tunnel.server1.remote in that way). In any case, VxLAN is a layer 2 technology. If you attach two tunnels to the same bridge then I would expect it to bridge between them.

If that’s not what you want, but instead you do want two different VxLANs / VNIs to connect to node 2, then how is an outbound packet from node 2 going to know which one to follow? One option is two different bridges, connected to two different virtual NICs as I explained. If that’s not an acceptable solution, please explain why it is not, and how you want it to work instead.

Another option might be to present the two VxLANs as two different 802.1q tags (subinterfaces on the same virtual NIC), but I don’t know if that’s doable without something more advanced like openvswitch.

bmullan · July 9, 2025, 4:40pm

First, it ts a VxLAN bridge.
Its also member of VNI 10 so it should only talk to other devices that are members of VNI 10 not just anyone.

re the example command:

incus network create brvxlan
tunnel.server1.protocol=vxlan
tunnel.server1.id=10
tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2
tunnel.server1.remote=10.0.1.3
ipv4.address=none
ipv6.address=none

The only info I could find talking about the Incus VxLAN type bridge was in the Documentation:

tunnel.NAME.group Multicast address for vxlan (used if local and remote aren’t set)
Key: tunnel.NAME.group
Type: string
Default: 239.0.0.1
Condition: vxlan

tunnel.NAME.id Specific tunnel ID to use for the vxlan tunnel

tunnel.NAME.interface Specific host interface to use for the tunnel
Key: tunnel.NAME.interface
Type: string
Default: •
Condition: vxlan

tunnel.NAME.local Local address for the tunnel (not necessary for multicast vxlan )

tunnel.NAME.port Specific port to use for the vxlan tunnel
Key: tunnel.NAME.port
Type: integer
Default: 0
Condition: vxlan

tunnel.NAME.protocol Tunneling protocol: vxlan or gre
Key: tunnel.NAME.protocol
Type: string
Default: •
Condition: standard mode

tunnel.NAME.remote Remote address for the tunnel (not necessary for multicast vxlan )
Key: tunnel.NAME.remote
Type: string
Default: •
Condition: gre or vxlan

tunnel.NAME.ttl Specific TTL to use for multicast routing topologies
Key: tunnel.NAME.ttl
Type: integer
Default: 1
Condition: vxlan

user.* User-provided free-form key/value pairs

Since in the above it does not address the questions, I first tried that example which creates the VxLAN Bridge. I was only guessing at the syntax (if it existed).

So I tried 2 possibilities just to see if by chance some syntax might work:
This was to see if you could have the Local tunnel IP and 2 “remote” Tunnel End Points (TEPs).

That did not work (but again I had no idea how to enter the command to even try it so I just tried:
on node1:

tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2 # if IP of Node 2 was 10.0.1.2
tunnel.server1.remote=10.0.1.3 # If IP of Node 3 was 10.0.1.2

So I wondered could that same config if it was possible to set the tunnel.id to both 10 & 100 thus letting any VM/Containers attached to that Incus VxLAN bridge talk to both VNI 10 & VNI 100?

I tried 2 diff config syntax but neither of these worked. But then again, the Docs don’t talk about if any of the command options can be used more than once:

tunnel.server1.id=10, 100
tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2
and
tunnel.server1.id=10
tunnel.server1.id=100
tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2

Lastly, as to your question:

If that’s not what you want, but instead you do want two different VxLANs / VNIs to connect to node 2, then how is an outbound packet from node 2 going to know which one to follow?

First, if Node 2 would be a member of both VNI 10 & 100.

In a VXLAN (Virtual eXtensible LAN) network, the Forwarding Database (FDB) is a table that stores mappings between MAC addresses and the corresponding VXLAN Tunnel Endpoints (VTEPs) or other interfaces.

So if a packet from Node1 arrives at Node2 … Node2’s FDB for VNI10 would get populated with the packet’s MAC address.

Next, in Node2 some VM/Container does something with the packet from Node1 and replies.

Node2 will lookup the now, destination MAC, in both the VNI 10 FDB and the VNI 100 FDB.

A match will be found in the VNI 10 FDB so to the return packet goes out that interface tagged with VNI 10 and Node1 gets its response.

candlerb · July 10, 2025, 9:59am

Well, according to the documentation, there’s no “vxlan bridge” network type in incus. It’s simply a “bridge”, to which you happen to have attached one or more vxlan interfaces. Those vxlan tunnels are regular port members of the bridge.

To demonstrate, here’s a bridge I created:

# incus network show brvxlan
config:
  tunnel.lab.id: "10"
  tunnel.lab.local: 10.12.255.13
  tunnel.lab.protocol: vxlan
  tunnel.lab.remote: 10.12.253.4
description: ""
name: brvxlan
type: bridge
used_by: []
managed: true
status: Created
locations:
- none
project: default

What this creates is a regular bridge. I can examine it using the usual tools:

root@nuc3:~# ip link show dev brvxlan
79: brvxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 10:66:6a:d5:54:f1 brd ff:ff:ff:ff:ff:ff

root@nuc3:~# ip link show master brvxlan
80: brvxlan-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue master brvxlan state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 46:fa:f2:15:7c:e6 brd ff:ff:ff:ff:ff:ff
81: brvxlan-lab: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master brvxlan state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 66:7a:5e:3a:5a:32 brd ff:ff:ff:ff:ff:ff

root@nuc3:~# bridge link | grep brvxlan
80: brvxlan-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1400 master brvxlan state forwarding priority 32 cost 100
81: brvxlan-lab: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 master brvxlan state forwarding priority 32 cost 100

Or the old-school way:

root@nuc3:~# brctl show brvxlan
bridge name     bridge id               STP enabled     interfaces
brvxlan         8000.10666ad554f1       no              brvxlan-lab
                                                        brvxlan-mtu

brvxlan-lab is the tunnel interface. (The reason for the second interface brvxlan-mtu is explained in the FAQ; it’s just to force the MTU down)

I can create two containers on the same host, and attach them to this bridge, and they can see each other (because it’s just a bridge). But, broadcasts are also seen encapsulated as UDP over the tunnel, using tcpdump. All as expected.

Now, I believe you’re saying that you want one bridge to talk to two different vxlan tunnels, but without bridging them together. As far as I can see, that’s not on offer here. A bridge bridges all its interfaces together. And even if it could distinguish the interfaces, how would you separate the traffic which you want to be able to traverse one vxlan interface but not the other?

The nearest I can think of is a Linux “VLAN-aware bridge”, which uses 802.1q VLAN tags to separate the traffic into different broadcast domains within a single bridge. However, you’d need to attach each vxlan interface to the bridge with a different tag. The vxlan link creation in incus doesn’t provide that option.

candlerb · July 10, 2025, 11:03am

P.S. Something I just learned the hard way.

The attribute tunnel.NAME.ttl is described as “Specific TTL to use for multicast routing topologies”. But in fact, it applies the TTL to all outgoing vxlan packets, including unicast; and it defaults to 1. (Tested with incus 6.0.4)

So if you want to send vxlan traffic through a router, you have to increase this to (at least) the number of hops required to reach the destination. Now I have:

root@nuc3:~# incus network show brvxlan
config:
  tunnel.lab.id: "10"
  tunnel.lab.local: 10.12.255.13
  tunnel.lab.protocol: vxlan
  tunnel.lab.remote: 10.12.253.4
  tunnel.lab.ttl: "2"
description: ""
name: brvxlan
type: bridge
used_by:
- /1.0/instances/test1
- /1.0/instances/test2
- /1.0/profiles/brvxlan-eth1
managed: true
status: Created
locations:
- none
project: default

and it’s happily bridging to a container connected to brvxlan on the remote server. (The remote bridge is configured identically, except local and remote are reversed)

root@nuc3:~# ip -d link show brvxlan-lab
115: brvxlan-lab: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master brvxlan state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 66:7a:5e:3a:5a:32 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 65535
    vxlan id 10 remote 10.12.253.4 local 10.12.255.13 srcport 0 0 dstport 8472 ttl 2 ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx
                                                                               ^^^^^
    bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8002 port_no 0x2 designated_port 32770 designated_cost 0 designated_bridge 8000.10:66:6a:d5:54:f1 designated_root 8000.10:66:6a:d5:54:f1 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

bmullan · July 10, 2025, 2:28pm

@candlerb
First, thanks for taking all this time looking into this!

Yes, I know a “bridge” is just a “bridge” but call me jaded after 30+ yr in networking.

side-note
But call me “jaded”, because 1 thing I’ve learned over those years is that Documentation nearly always has things left unsaid.

That’s why voice conversations resolve questions like this so much faster but here we are.

There actually is a term for Engineers & Tech Doc writers called MEGO (My Eyes Glaze Over).
Meaning some Developer/Tech Writer who is “expert” in something they created and trying to explain/document it often “glaze over” mentioning some “info” that the Developer, who works with it every day, just “assumes” everyone would know or would assume.

In my life I’ve written many tech docs too and I’m as bad at that as anyone.

But the fact is that the consumers of that “info” are usually not “experts” in whatever that “thing” is. And since they aren’t, they don’t even know there was something not said in the Docs.

But back to the topic…

When I able to successfully execute:

On each Node I create a VXLAN bridge named brvxlan using this general command:

incus network create brvxlan
tunnel.server1.protocol=vxlan
tunnnel.server1.id=10
tunnel.server1.local=10.0.1.1
tunnel.server1.remote=10.0.1.2
tunnel.server1.remote=10.0.1.3
ipv4.address=none
ipv6.address=none

I guess I was just assuming a MEGO moment in the docs.

Incus VLAN interface participates in segmenting a single physical network while an Incus VXLAN interface participates in extending Layer 2 networks across often disparate physical networks

Also… yeah I know vxlan would be lowering MTU to 1450

I guess I was reading way to much into the Docs and started thinking since Incus supports BGP that perhaps configuring an Incus Bridge VxLAN interface was somehow enabling IRB.

IRB (Integrated Routing and Bridging) and VXLAN are distinct but related technologies.

IRB allows a network device to perform both Layer 2 Bridging and Layer 3 Routing, enabling communication between different subnets within the same VXLAN fabric.

Guess I was hoping it was doing that …

Anyway… again thanks and like everything in linux/networking there are always more than one way to skin a cat. I’ve been meaning to look into Geneve as alternative to VxLAN anyway.

So this gives me the excuse/chance to learn Geneve now.