Routed NIC change default gateway metric

Hello,

I have an instance with two NIC’s configured:

  eth0:
    name: eth0
    network: ovntest
    type: nic
  eth1:
    ipv4.address: <home lan ip>
    name: eth1
    nictype: routed
    parent: enp10s0
    type: nic

After starting the instance the routing table looks like this:

default via 169.254.0.1 dev eth1 
default via 10.43.146.1 dev eth0 proto dhcp src 10.43.146.2 metric 1024 
10.43.146.0/24 dev eth0 proto kernel scope link src 10.43.146.2 metric 1024 
10.43.146.1 dev eth0 proto dhcp scope link src 10.43.146.2 metric 1024 
10.102.89.1 via 10.43.146.1 dev eth0 proto dhcp src 10.43.146.2 metric 1024 
169.254.0.1 dev eth1 scope link 

Which means all “unknown” traffic will be routed through eth1. In general this is fine but if you have OVN peering configured you can’t reach the peered OVN because of the lower metric (higher priority) of eth1.

Changing the order of the NIC’s doesn’t change the behaviour nor trying to influence network settings by editing “/etc/systemd/network/eth1.network” as the NIC is created during instance start. Reviewing the network options allows to set “ipv4.gateway to none” which results into the following table:

default via 10.43.146.1 dev eth0 proto dhcp src 10.43.146.2 metric 1024 
10.43.146.0/24 dev eth0 proto kernel scope link src 10.43.146.2 metric 1024 
10.43.146.1 dev eth0 proto dhcp scope link src 10.43.146.2 metric 1024 
10.102.89.1 via 10.43.146.1 dev eth0 proto dhcp src 10.43.146.2 metric 1024 

No route at all added as it is disabled. However, adding it back manual by editing “/etc/systemd/network/eth1.network” didn’t have the desired result either.

My question is how can I influence the instance routing table to give eth0 the lower metric to reach other peered OVN networks?

After a bit more testing I found a possible workaround by chnaging the “default” route to a “network” route:

ip route del default via 169.254.0.1
ip route add <home network>/24 via 169.254.0.1

Which results into the following routing table:

default via 10.43.146.1 dev eth0 proto dhcp src 10.43.146.2 metric 1024 
10.43.146.0/24 dev eth0 proto kernel scope link src 10.43.146.2 metric 1024 
10.43.146.1 dev eth0 proto dhcp scope link src 10.43.146.2 metric 1024 
10.102.89.1 via 10.43.146.1 dev eth0 proto dhcp src 10.43.146.2 metric 1024 
169.254.0.1 dev eth1 scope link 
<home network>/24 via 169.254.0.1 dev eth1 

Now the instance is able to reach out to all networks including peered OVN’s and you can access it from the home network.

To automate the route change during instance start it requires to place a hook into the network initialisation. For a Bookworm container it is “networkd-dispatcher” which allows to execute scripts. Placing a script at “/etc/networkd-dispatcher/routable.d/route_change_script.sh”

#!/bin/bash

if [[ $IFACE == "eth1" && $AdministrativeState == "unmanaged" ]]; then
    GW_IP=$(ip route show |grep default|grep eth1 |cut -d ' ' -f 3)

    if [[ $GW_IP == "169.254.0.1" ]]; then
        ip route del default via 169.254.0.1
        ip route add <home network>/24 via 169.254.0.1
    fi
fi

exit 0

Does exactly what is required during startup or restart.

As mentioned above this is a workaround but I would prefer a more dynamic solution managed by Incus configuration settings.

Any useful input is welcome…