Incus Cluster Network Issue

Incus
,
1

I’m currently excitedly transitioning from Proxmox to Incus for my homelab, and I’ve encountered a challenge with network configuration for which I’d deeply appreciate your expertise. I just spend the better part of my Sunday trying to get my cluster up and running on 7 hosts. I successfully launched a container with the following command:

incus launch images:debian/12 pihole --config limits.cpu=1 --config limits.memory=512MiB --target=rpi3bp

The container is running well. However, I’m struggling to configure the network with a static IP. My aim is simple: to bridge the host network to the container, ensuring it’s part of the same subnet (192.168.1.1/24).

The closest I’ve come to a working setup involved these steps:

incus network create ibr0 --target=t130
incus network create ibr0 --target=l640
incus network create ibr0 --target=rpi3bp
incus network create ibr0 --target=k72f
incus network create ibr0 --target=ux390ua
incus network create ibr0 --target=1018p
incus network create ibr0

followed by:

incus network set ibr0 ipv4.address=192.168.1.254/24
incus network attach ibr0 pihole eth0

I managed to receive an IP from my DHCP server:

root@rpi3bp:~# incus list
±--------±--------±---------------------±-----±----------±----------±---------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
±--------±--------±---------------------±-----±----------±----------±---------+
| pihole | RUNNING | 192.168.1.165 (eth0) | | CONTAINER | 0 | rpi3bp |
±--------±--------±---------------------±-----±----------±----------±---------+

Setting up the static IP:

incus stop pihole
incus network detach ibr0 pihole
incus network attach ibr0 pihole eth0
incus config device set pihole eth0 ipv4.address=192.168.1.4
incus start pihole

The container can’t reach my local network or the Internet:

root@pihole:~# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 192.168.1.4 icmp_seq=1 Destination Host Unreachable
From 192.168.1.4 icmp_seq=2 Destination Host Unreachable
From 192.168.1.4 icmp_seq=3 Destination Host Unreachable

root@pihole4:~# ip route
default via 192.168.1.254 dev eth0 proto dhcp src 192.168.1.4 metric 1024
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.4 metric 1024
192.168.1.254 dev eth0 proto dhcp scope link src 192.168.1.4 metric 1024

I also tried following a guide on bridge networking in LXD (How to Configure Bridge Networking in LXD | Sean Blanchfield), which worked partially (I was able to get traffic in and out of the container) but didn’t allow setting a static IP because the network bridge was unmanaged.

If anyone can offer guidance on this seemingly straightforward yet challenging setup, I would be immensely grateful. Thank you in advance for your assistance!

2

If the goal is to share the physical network of the host, then your best bet is to not create a bridge through Incus at all but instead configure your host’s networking to provide you with a usable bridge.

On Ubuntu you’d do that through /etc/netplan, on Debian, through /etc/network/interfaces.

In both cases, your goal will be to get the physical network card of your server and have it be part of a bridge, then make sure that no IP configuration remains on the physical card itself but that everything is done through the bridge instead.

Once that works, you can simply configure Incus to place your instances on that bridge with:

incus profile device add default eth0 nic nictype=bridged parent=BR-XYZ name=eth0
3

Thank you for the assistance Stéphane!

Here is my current configuration:
/etc/network/interfaces:

iface eth0 inet manual

auto incusbr0
iface incusbr0 inet static
address 192.168.1.3
netmask 255.255.255.0
gateway 192.168.1.1
bridge-ifaces eth0
bridge-ports eth0
up ip link set eth0 up

Adding the bridge to the default profile:

incus profile device add default eth0 nic nictype=bridged parent=incusbr0 name=eth0

Starting the instance now puts it on the right subnet and I’m able to get traffic in and out - perfect. Now the last step is to configure the static IP, but I’m having issues with getting it to work properly.

root@rpi3bp:~# incus config device set pihole eth0 ipv4.address 192.168.1.4
Error: Device from profile(s) cannot be modified for individual instance. Override device or modify profile instead

root@rpi3bp:~# incus config device override pihole eth0
root@rpi3bp:~# incus config device set pihole eth0 ipv4.address=192.168.1.4
Error: Invalid devices: Device validation failed for “eth0”: Cannot use manually specified ipv4.address when using unmanaged parent bridge

I can pass this command:

incus config device set pihole eth0 ipv4.address=192.168.1.4 security.ipv4_filtering=true

But I’m unable to get the static IPv4 address on the container:

root@rpi3bp:~# incus list
±--------±--------±-----±--------------------------------------------±----------±----------±---------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
±--------±--------±-----±--------------------------------------------±----------±----------±---------+
| pihole | RUNNING | | 2a01:e0a:4af:c7a0:216:3eff:fe06:721b (eth0) | CONTAINER | 0 | rpi3bp |
±--------±--------±-----±--------------------------------------------±----------±----------±---------+

4

Because your external router/DHCP is in charge of IP assignment, you can’t use ipv4.address in Incus.

Instead you either need to define static DHCP leases in your router/DHCP server to mimic what Incus would normally do for you when you set ipv4.address, or you need to instead put in static networking settings in the instance itself (its /etc/network/interfaces or equivalent).

5

Thanks Stéphane - problem solved!

For others interested in configuring a static IP on the debian12 image, I changed the following file on the instance: /etc/systemd/network/eth0.network

[Match]
Name=eth0

[Network]
Address=192.168.1.4/24
Gateway=192.168.1.1

And then:

restart the systemctl restart systemd-networkd

1 Like