IPv6 DHCPv6-PD per-node different prefixes in Incus cluster — how to handle cross-node connectivity?

Incus
,
1

Hello,

I have an Incus cluster where all nodes receive IPv6 via DHCPv6 Prefix Delegation (PD /96) on the default incusbr0 bridge. The problem is that each node gets a different /96 prefix from the DHCPv6 server, which causes two issues:

  • Instances on different nodes cannot reach each other directly over IPv6
  • I cannot live migrate instances between nodes since the IPv6 addressing changes

What is the best approach to solve this in an Incus cluster setup?

2

Are you sure about the server assigning prefixes?

Do you setup the node with incus admin init? If so you could paste a yaml (you can generate one with that) to have the same prefix on all nodes.

3 
config:
  cluster.https_address: 192.168.0.101:8443
  core.https_address: 192.168.0.101:8443
  core.https_trusted_proxy: 127.0.0.1,192.168.0.101,192.168.0.3
  oidc.audience: account
  oidc.claim: email
  oidc.client.id: incus
  oidc.issuer: https://auth.sy.sa/realms/master
  oidc.scopes: openid,profile,email
  user.ui.sso_only: "true"
  user.ui.title: Salem Infra
networks:
  - config:
      ipv4.address: 10.88.223.1/24
      ipv4.nat: "true"
      ipv6.address: 2001:678:1120:0:1:100:0:1/96
      ipv6.dhcp: "true"
      ipv6.dhcp.stateful: "true"
      ipv6.nat: "false"
    description: ""
    name: incusbr0
    type: bridge
    project: default
  - config: {}
    description: ""
    name: macvlan
    type: macvlan
    project: default
storage_pools:
  - config: {}
    description: ""
    name: local
    driver: dir
storage_volumes: []
profiles:
  - config: {}
    description: Default Incus profile
    devices:
      eth0:
        name: eth0
        network: macvlan
        type: nic
      root:
        path: /
        pool: local
        type: disk
    name: default
    project: ""
projects:
  - config:
      features.images: "true"
      features.profiles: "true"
      features.storage.buckets: "true"
      features.storage.volumes: "true"
    description: ""
    name: Authentication
  - config:
      features.images: "true"
      features.networks: "true"
      features.networks.zones: "true"
      features.profiles: "true"
      features.storage.buckets: "true"
      features.storage.volumes: "true"
    description: Default Incus project
    name: default
certificates: []
cluster_groups: []

nodes subnets are:

  • nodes1: 2001:678:1120:0:1:100::/96
  • nodes2: 2001:678:1120:0:1:102::/96
  • nodes3: 2001:678:1120:0:1:103::/96
  • nodes4: 2001:678:1120:0:1:104::/96

and container ipv6 is 2001:678:1120:0:1:100:cd75:c19b