Dual NICs: eth0 on br-in (managed), eth1 on br-ex (external unmanaged bridge)

Incus
, ,
1

Hi guys,

I’ve installed Incus on my Ubuntu 24.04 (through system packages) and configured the following Netplan:

$ netplan status --all

     Online state: online
    DNS Addresses: 127.0.0.53 (stub)
       DNS Search: .

●  1: lo ethernet UNKNOWN/UP (unmanaged)
      MAC Address: 00:00:00:00:00:00
        Addresses: 127.0.0.1/8
                   ::1/128

●  2: enp0s31f6 ethernet UP (networkd: enp0s31f6)
      MAC Address: 48:4d:7e:fb:ee:be (Intel Corporation)
           Bridge: br-ex

●  3: br-ex bridge UP (networkd: br-ex)
      MAC Address: 42:e5:16:42:ab:b3
        Addresses: 192.168.18.100/24
                   fe80::40e5:16ff:fe42:abb3/64 (link)
    DNS Addresses: 8.8.8.8
                   1.1.1.1
           Routes: default via 192.168.18.1 (static)
                   192.168.18.0/24 from 192.168.18.100 (link)
                   fe80::/64 metric 256
       Interfaces: enp0s31f6

●  4: br-in bridge DOWN/UP (unmanaged)
      MAC Address: 00:16:3e:e3:19:d4
        Addresses: 10.82.109.1/24
                   fd42:488d:f93d:3a76::1/64
                   fe80::216:3eff:fee3:19d4/64 (link)
           Routes: 10.82.109.0/24 from 10.82.109.1 (link)
                   fd42:488d:f93d:3a76::/64 metric 256
                   fe80::/64 metric 256

That is, I’ve created a br-ex bridge that has the enp0s31f6 as it’s interface. This way, I can use this bridge as the network and the instances get an IP from my LAN.

The bridge created by Incus is the br-in. What I want is: instances with NIC devices eth0 and eth1 for br-in and br-ex, respectively, each one with an IP (from DHCP or set manually).

I did some research and found this:

On this topic, @stgraber says that only eth0 gets IP from DHCP and all the rest of interfaces are ignored.

Is there any way to get this result? Should I use OVN for that?

2

You misunderstood, he means only eth0 is preset to use dhcp. Other ports don’t have any config preset. You can manually config other network port according to the instance’s network manager’s instruction.

3

At the time this thread was opened there was no other option to get a second NIC working with DHCP or similar. As far as I remember this has been fixed / improved in one of the later Incus versions Default routes and multiple DHCP clients for OCI containers.

Have you tested to assign a second NIC device to an OCI if it retrieves an IP? It works for me on latest Incus 6.21 version.

4

Hi,

I reinstalled Incus to use Zabbly repository source, and now my Incus is on version 6.21 as well. To try what you suggested I’ve created a profile with two interfaces:

# dualnet.yaml
config: {}
description: Dual network profile with eth0=br-ex, eth1=br-in (unmanaged)
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br-ex
    type: nic
  eth1:
    name: eth1
    nictype: bridged
    parent: br-in
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: dualnet
used_by:
- /1.0/instances/c3
- /1.0/instances/c4
project: default

So I created a instance with this profile: incus launch images:ubuntu/24.04 c4 -p dualnet.

Then I got IPs from the interfaces but the second interface only gets an IPv6 address and not a IPv4 one.

$ incus list
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4          |                      IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+
| c4   | RUNNING | 192.168.18.121 (eth0) | fd42:872c:3f50:233c:1266:6aff:fed3:3e65 (eth1) | CONTAINER | 0         |
|      |         |                       | 2804:9f8:478b:aa00:1266:6aff:fee9:6dd0 (eth0)  |           |           |
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+

For the record, I didn’t configured anything inside the container, as I expect to work “out-of-box”. Also, maybe it would be interesting to show my Netplan config itself, if something is missing:

network:
  version: 2
  ethernets:
    enp0s31f6:
      dhcp4: false
  bridges:
    br-ex:
      interfaces:
        - enp0s31f6
      addresses:
      - "192.168.18.100/24"
      nameservers:
        addresses:
        - 8.8.8.8
        - 1.1.1.1
        search: []
      routes:
      - to: "default"
        via: "192.168.18.1"

How did you launch the instances to get IPv4 addresses from both NICs? Is there something I’m missing here?

5

You need to config eth1 to use dhcp in instances, as I said other ports don’t have any config preset, so they will not work out-of-box.

6

Clean start is always good, makes it much easier to debug.

Given that you get an IPv6 means your network setup is working in general. Unfortunately IPv4 works a bit different compared to IPv6 to retrieve an IP.

Unfortunately all images provided from images.linuxcontainer.org only have the default eth0 configured to retrieve IPv4 over DHCP. You need to either add a cloud-init section to your profile or image or you go with static IPs.

One issue you are facing with two NIC’s is which one should be used for the default route. From what I remember the second one wins with default settings. So I decided to go with static IP’s on my second interface but your use case or milage might vary.

For a test try to add a static IP to the second interface:

incus config device override c4 eth1 ipv4.address=10.82.109.x/24

Just replace x with a free IP from that network.

In case of cloud-init you might try:

cat <<EOF | incus profile set dualnet cloud-init.network-config -
network:
version: 2
ethernets:
eth1:
dhcp4: true
EOF

This should add the defined cloud-init to your instance. I requires to launch a cloud-init aware instance which doesn’t really work with OCI.