How can I use BGP with incus cluster?

Incus
,
21

Hm. I have to idea. Any suggestions?

22

Can you show your updated:

  • incus network show NAME
  • incus config show --expanded NAME
  • incus query /internal/debug/bgp (from the machine currently running the instance)
23 
+-------+---------+-----------------------+------+-----------------+-----------+----------+
| NAME  |  STATE  |         IPV4          | IPV6 |      TYPE       | SNAPSHOTS | LOCATION |
+-------+---------+-----------------------+------+-----------------+-----------+----------+
| debug | RUNNING | 10.35.28.245 (enp5s0) |      | VIRTUAL-MACHINE | 0         | mini2    |
+-------+---------+-----------------------+------+-----------------+-----------+----------+

vm in mini2 server

ansible@mini2 ~> sudo incus network show incusbr0
config:
  bgp.peers.debug.address: 192.168.20.51
  bgp.peers.debug.asn: "64512"
  bgp.peers.mikrotik.address: 192.168.20.1
  bgp.peers.mikrotik.asn: "65540"
  ipv4.address: 10.35.28.1/24
  ipv4.nat: "false"
  ipv6.address: none
description: ""
name: incusbr0
type: bridge
used_by:
- /1.0/instances/debug
- /1.0/profiles/default
- /1.0/profiles/packer
managed: true
status: Created
locations:
- mini1
- mini3
- worker1
- mini2
- mini5
- mini4
- lb
project: default

ansible@mini2 ~> sudo incus config show --expanded debug
architecture: x86_64
config:
  cluster.evacuate: auto
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (cloud) (20251126_07:42)
  image.name: ubuntu-noble-amd64-cloud-20251126_07:42
  image.os: ubuntu
  image.release: noble
  image.serial: "20251126_07:42"
  image.variant: cloud
  limits.cpu: "2"
  limits.memory: 2GiB
  migration.stateful: "true"
  volatile.base_image: 8d3e6ff9770f2d2da831982c06967600e963ca0eb58353b2bf1f6c3b8b736b24
  volatile.cloud-init.instance-id: 517f6db5-b2df-4f1b-a90b-5f5a092822c2
  volatile.eth0.host_name: tap1367dc15
  volatile.eth0.hwaddr: 10:66:6a:25:38:bb
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: eefc17b4-cd07-4feb-942b-79c23c664df2
  volatile.uuid.generation: eefc17b4-cd07-4feb-942b-79c23c664df2
  volatile.vm.definition: pc-q35-10.1
  volatile.vm.rtc_adjustment: "-1"
  volatile.vm.rtc_offset: "0"
  volatile.vsock_id: "1410789558"
devices:
  eth0:
    ipv4.routes.external: 10.35.28.218/32
    network: incusbr0
    type: nic
  root:
    path: /
    pool: local
    size: 20GiB
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

ansible@mini2 ~> sudo incus query /internal/debug/bgp
{
	"peers": [
		{
			"address": "192.168.20.1",
			"asn": 65540,
			"count": 1,
			"holdtime": 0,
			"password": ""
		},
		{
			"address": "192.168.20.51",
			"asn": 64512,
			"count": 1,
			"holdtime": 0,
			"password": ""
		}
	],
	"prefixes": [
		{
			"nexthop": "0.0.0.0",
			"owner": "network_2",
			"prefix": "10.35.28.0/24"
		},
		{
			"nexthop": "0.0.0.0",
			"owner": "instance_19_eth0",
			"prefix": "10.35.28.218/32"
		}
	],
	"server": {
		"address": "0.0.0.0:179",
		"asn": 65536,
		"router_id": "192.168.20.22",
		"running": true
	}
}

/32 prefix exists.

And last problem - how to automate it?

24

Okay, so that’s looking good. By the way, not that the ipv4.routes.external doesn’t match the instance’s address so that is likely broken right now.

It should be pretty easy to script it, effectively:

  • Get the instance list

  • Check if the instance NIC already has ipv4.address and ipv4.routes.external set

  • If not, get the instance current IPv4 address and put it in both config keys

We can’t do too much more than that automatically in Incus as we’re not in charge of the instance’s IP address, it’s automatically allocated by dnsmasq and we’re not getting notified or anything when the lease is made, or changed.

If you’re using tooling like Terraform/OpenTofu to create the instances, you could have that set both config keys for you as part of creation.