Thanks joen, I manually created vmbr0 bridges on each host node and somehow managed to get the nodes and cluster to recognise it. As explained above, I want to work with my Proxmox cluster, and it uses a default bridge of vmbr0. I have removed all firewall rules from hosts and guests. Some configs follow…
cos0 ~ incus cluster list
+------+-------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS | MESSAGE |
+------+-------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cos0 | https://cos0:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+------+-------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cos1 | https://cos1:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+------+-------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cos2 | https://cos2:8443 | database | x86_64 | default | | ONLINE | Fully operational |
+------+-------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
cos0 ~ incus network list # baremetal Minisform MS-01
+-------------+----------+---------+------+------+-------------+---------+-------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+-------------+----------+---------+------+------+-------------+---------+-------+
| enp2s0f0np0 | physical | NO | | | | 0 | |
+-------------+----------+---------+------+------+-------------+---------+-------+
| enp2s0f1np1 | physical | NO | | | | 0 | |
+-------------+----------+---------+------+------+-------------+---------+-------+
| enp87s0 | physical | NO | | | | 0 | |
+-------------+----------+---------+------+------+-------------+---------+-------+
| enp89s0 | physical | NO | | | | 0 | |
+-------------+----------+---------+------+------+-------------+---------+-------+
| vmbr0 | bridge | NO | | | | 6 | |
+-------------+----------+---------+------+------+-------------+---------+-------+
| wlan0 | physical | NO | | | | 0 | |
+-------------+----------+---------+------+------+-------------+---------+-------+
cos1 ~ incus network list # Promox VM (same for cos2)
+---------+----------+---------+------+------+-------------+---------+-------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+---------+----------+---------+------+------+-------------+---------+-------+
| enp6s18 | physical | NO | | | | 0 | |
+---------+----------+---------+------+------+-------------+---------+-------+
| vmbr0 | bridge | NO | | | | 6 | |
+---------+----------+---------+------+------+-------------+---------+-------+
cos0 ~ incus info local:
config:
cluster.https_address: cos0:8443
core.https_address: cos0:8443
api_extensions:
- storage_zfs_remove_snapshots
- etc ...
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: markc
auth_user_method: unix
environment:
addresses:
- cos0:8443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
etc ...
-----END CERTIFICATE-----
certificate_fingerprint: 4fcb...
driver: lxc | qemu
driver_version: 6.0.0 | 9.0.0
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
idmapped_mounts: "true"
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
uevent_injection: "true"
unpriv_binfmt: "true"
unpriv_fscaps: "true"
kernel_version: 6.9.0-1-cachyos
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: CachyOS Linux
os_version: ""
project: default
server: incus
server_clustered: true
server_event_mode: full-mesh
server_name: cos0
server_pid: 1496
server_version: "6.1"
storage: zfs
storage_version: 2.2.3-14_g228ff3867
storage_supported_drivers:
- name: zfs
version: 2.2.3-14_g228ff3867
remote: false
- name: btrfs
version: 6.8.1
remote: false
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.23(2) (2023-11-21) / 1.02.197 (2023-11-21) / 4.48.0
remote: false
cos1 ~ inus info local: # same as cos2
config:
cluster.https_address: cos1:8443
core.https_address: cos1:8443
api_extensions:
- storage_zfs_remove_snapshots
- etc ...
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: sysadm
auth_user_method: unix
environment:
addresses:
- cos1:8443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
etc ...
-----END CERTIFICATE-----
certificate_fingerprint: 4fcb...
driver: lxc | qemu
driver_version: 6.0.0 | 9.0.0
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
idmapped_mounts: "true"
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
uevent_injection: "true"
unpriv_binfmt: "true"
unpriv_fscaps: "true"
kernel_version: 6.8.9-4-cachyos
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: CachyOS Linux
os_version: ""
project: default
server: incus
server_clustered: true
server_event_mode: full-mesh
server_name: cos1
server_pid: 2864
server_version: "6.1"
storage: zfs
storage_version: 2.2.3-14_g228ff3867
storage_supported_drivers:
- name: zfs
version: 2.2.3-14_g228ff3867
remote: false
- name: btrfs
version: 6.8.1
remote: false
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.23(2) (2023-11-21) / 1.02.197 (2023-11-21) / 4.48.0
remote: false
- name: lvmcluster
version: 2.03.23(2) (2023-11-21) / 1.02.197 (2023-11-21) / 4.48.0
remote: true
cos0 ~ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7a:49:c4:61:cc:60 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.234/24 brd 192.168.1.255 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 2403:580a:5e99:0:7849:c4ff:fe61:cc60/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 597sec preferred_lft 597sec
inet6 fe80::7849:c4ff:fe61:cc60/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
3: enp87s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 58:47:ca:77:0c:28 brd ff:ff:ff:ff:ff:ff
4: enp89s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr0 state DOWN group default qlen 1000
link/ether 58:47:ca:77:0c:29 brd ff:ff:ff:ff:ff:ff
5: enp2s0f0np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr0 state DOWN group default qlen 1000
link/ether 58:47:ca:77:0c:26 brd ff:ff:ff:ff:ff:ff
6: enp2s0f1np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr0 state DOWN group default qlen 1000
link/ether 58:47:ca:77:0c:27 brd ff:ff:ff:ff:ff:ff
7: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 04:f4:d8:18:3c:c3 brd ff:ff:ff:ff:ff:ff
9: veth1ff771d2@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether 4a:68:ef:c0:6a:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
11: veth0607bdbd@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether 42:74:f5:5c:b8:16 brd ff:ff:ff:ff:ff:ff link-netnsid 1
cos0 ~ ip r
default via 192.168.1.1 dev vmbr0 proto static
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.234
cos1 ~ ip a # same/similar for cos2
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp6s18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vmbr0 state UP group default qlen 1000
link/ether bc:24:11:c4:66:dc brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 1a:63:15:80:49:6d brd ff:ff:ff:ff:ff:ff
inet 192.168.1.208/24 brd 192.168.1.255 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 2403:580a:5e99:0:1863:15ff:fe80:496d/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 594sec preferred_lft 594sec
inet6 fe80::1863:15ff:fe80:496d/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
7: veth04adbaa6@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether 56:b1:dd:a5:62:9e brd ff:ff:ff:ff:ff:ff link-netnsid 2
9: vethc0814fda@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether 56:2d:6b:cc:97:3b brd ff:ff:ff:ff:ff:ff link-netnsid 3
cos1 ~ ip r
default via 192.168.1.1 dev vmbr0 proto static
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.208
As noted, iptables, nftables and ebtables lists are all empty.