Hello,
I am having trouble with network forwarding in Incus. I set up a bridge network and the corresponding child OVN network, then created a network forward to one of my instances. After a lot of debugging and searching, I found a thread that mentioned ARP requests aren’t being replied to. I can reproduce the issue on a single instance (no cluster). It looks like this might be a regression in OVN that was fixed before but seems to have come back.
This was all tested on a new Ubuntu 24.04 VM using the zabbly/incus and stgraber/ovn-stable PPAs.
username@incus-test:~$ incus --version
6.15
username@incus-test:~$ ovs-vsctl --version
ovs-vsctl (Open vSwitch) 3.4.0
DB Schema 8.7.0
username@incus-test:~$ ovn-nbctl --version
ovn-nbctl 24.03.3
Open vSwitch Library 3.4.0
DB Schema 7.3.0
username@incus-test:~$ ovn-sbctl --version
ovn-sbctl 24.03.3
Open vSwitch Library 3.4.0
DB Schema 20.33.0
My network config:
username@incus-test:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:d5:15:36 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.125/24 brd 192.168.122.255 scope global dynamic noprefixroute enp1s0
valid_lft 2645sec preferred_lft 2645sec
inet6 fe80::5054:ff:fed5:1536/64 scope link
valid_lft forever preferred_lft forever
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 56:54:21:81:fa:50 brd ff:ff:ff:ff:ff:ff
4: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 22:ae:8e:46:46:77 brd ff:ff:ff:ff:ff:ff
5: incusbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 10:66:6a:df:13:7d brd ff:ff:ff:ff:ff:ff
inet 100.64.0.1/11 brd 100.95.255.255 scope global incusbr0
valid_lft forever preferred_lft forever
21: incusovn1b@incusovn1a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether 3a:d7:cb:37:1f:59 brd ff:ff:ff:ff:ff:ff
22: incusovn1a@incusovn1b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP group default qlen 1000
link/ether 4a:9a:34:b6:73:71 brd ff:ff:ff:ff:ff:ff
23: incusovn1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 32:0d:15:26:d0:43 brd ff:ff:ff:ff:ff:ff
27: veth2a75932d@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
link/ether b2:59:e9:d6:df:0b brd ff:ff:ff:ff:ff:ff link-netnsid 0
username@incus-test:~$ ip route
default via 192.168.122.1 dev enp1s0 proto dhcp src 192.168.122.125 metric 100
100.64.0.0/11 dev incusbr0 proto kernel scope link src 100.64.0.1
100.96.0.0/11 dev incusbr0 proto static scope link
192.168.122.0/24 dev enp1s0 proto kernel scope link src 192.168.122.125 metric 100
Incus network config:
username@incus-test:~$ incus network ls
+------------+----------+---------+---------------+------+-------------+---------+---------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+------------+----------+---------+---------------+------+-------------+---------+---------+
| br-int | bridge | NO | | | | 0 | |
+------------+----------+---------+---------------+------+-------------+---------+---------+
| enp1s0 | physical | NO | | | | 0 | |
+------------+----------+---------+---------------+------+-------------+---------+---------+
| incusbr0 | bridge | YES | 100.64.0.1/11 | none | | 1 | CREATED |
+------------+----------+---------+---------------+------+-------------+---------+---------+
| incusovn1 | bridge | NO | | | | 0 | |
+------------+----------+---------+---------------+------+-------------+---------+---------+
| incusovn1a | unknown | NO | | | | 0 | |
+------------+----------+---------+---------------+------+-------------+---------+---------+
| incusovn1b | unknown | NO | | | | 0 | |
+------------+----------+---------+---------------+------+-------------+---------+---------+
| lo | loopback | NO | | | | 0 | |
+------------+----------+---------+---------------+------+-------------+---------+---------+
| ovs-system | unknown | NO | | | | 0 | |
+------------+----------+---------+---------------+------+-------------+---------+---------+
username@incus-test:~$ incus network ls --project lab-1
+------------+------+---------+----------------+------+-------------+---------+---------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+------------+------+---------+----------------+------+-------------+---------+---------+
| n-4-uplink | ovn | YES | 192.168.0.1/24 | none | | 1 | CREATED |
+------------+------+---------+----------------+------+-------------+---------+---------+
Incus network details:
username@incus-test:~$ incus network show incusbr0
config:
bridge.mtu: "1500"
dns.mode: none
dns.nameservers: 8.8.8.8,8.8.4.4,1.1.1.1,1.0.0.1
ipv4.address: 100.64.0.1/11
ipv4.dhcp.ranges: 100.64.0.2-100.64.0.2
ipv4.nat: "true"
ipv4.ovn.ranges: 100.64.0.3-100.95.255.254
ipv4.routes: 100.96.0.0/11
ipv4.routing: "true"
ipv6.address: none
description: ""
name: incusbr0
type: bridge
used_by:
- /1.0/networks/n-4-uplink?project=lab-1
managed: true
status: Created
locations:
- none
project: default
username@incus-test:~$ incus network show n-4-uplink --project lab-1
config:
bridge.mtu: "1500"
dns.nameservers: 8.8.8.8,8.8.4.4,1.1.1.1,1.0.0.1
ipv4.address: 192.168.0.1/24
ipv4.nat: "true"
ipv6.address: none
network: incusbr0
volatile.network.ipv4.address: 100.64.0.3
description: ""
name: n-4-uplink
type: ovn
used_by:
- /1.0/instances/i-4-1?project=lab-1
managed: true
status: Created
locations:
- none
project: lab-1
username@incus-test:~$ incus network forward show n-4-uplink 100.96.0.10 --project lab-1
description: ""
config: {}
ports:
- description: ""
protocol: tcp
listen_port: "1024"
target_port: "80"
target_address: 192.168.0.2
snat: false
listen_address: 100.96.0.10
location: ""
Running curl 100.96.0.10:1024 results in the following trace:
username@incus-test:~$ sudo tcpdump -i incusbr0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on incusbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:42:50.526354 ARP, Request who-has 100.96.0.10 tell incus-test, length 28
21:42:51.542105 ARP, Request who-has 100.96.0.10 tell incus-test, length 28
21:42:52.566009 ARP, Request who-has 100.96.0.10 tell incus-test, length 28
^C <killed after several minutes>
3 packets captured
3 packets received by filter
0 packets dropped by kernel
Using the default firewall rules:
table inet filter {
chain input {
type filter hook input priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy accept;
}
chain output {
type filter hook output priority filter; policy accept;
}
}
table inet incus {
chain pstrt.incusbr0 {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 100.64.0.0/11 ip daddr != 100.64.0.0/11 masquerade
}
chain fwd.incusbr0 {
type filter hook forward priority filter; policy accept;
ip version 4 oifname "incusbr0" accept
ip version 4 iifname "incusbr0" accept
}
chain in.incusbr0 {
type filter hook input priority filter; policy accept;
iifname "incusbr0" tcp dport 53 accept
iifname "incusbr0" udp dport 53 accept
iifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
iifname "incusbr0" udp dport 67 accept
iifname "incusbr0" ip protocol udp udp checksum set 0
}
chain out.incusbr0 {
type filter hook output priority filter; policy accept;
oifname "incusbr0" tcp sport 53 accept
oifname "incusbr0" udp sport 53 accept
oifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
oifname "incusbr0" udp sport 67 accept
oifname "incusbr0" ip protocol udp udp checksum set 0
}
}
ovs/ovn network details:
username@incus-test:~$ sudo ovs-vsctl show
4e60ff7f-7a18-4e54-82d1-b248490a6032
Bridge br-int
fail_mode: secure
datapath_type: system
Port br-int
Interface br-int
type: internal
Port veth2a75932d
Interface veth2a75932d
Port patch-br-int-to-incus-net5-ls-ext-lsp-provider
Interface patch-br-int-to-incus-net5-ls-ext-lsp-provider
type: patch
options: {peer=patch-incus-net5-ls-ext-lsp-provider-to-br-int}
Bridge incusovn1
Port incusovn1
Interface incusovn1
Port incusovn1b
Interface incusovn1b
Port patch-incus-net5-ls-ext-lsp-provider-to-br-int
Interface patch-incus-net5-ls-ext-lsp-provider-to-br-int
type: patch
options: {peer=patch-br-int-to-incus-net5-ls-ext-lsp-provider}
ovs_version: "3.4.0"
username@incus-test:~$ sudo ovn-nbctl show
switch 61d9133c-d8d0-4662-854f-92dcb20c692c (incus-net5-ls-int)
port incus-net5-instance-639da7f5-e0a2-47e1-8e43-b016d8450378-eth0
addresses: ["10:66:6a:25:fa:fc 192.168.0.2"]
port incus-net5-ls-int-lsp-router
type: router
router-port: incus-net5-lr-lrp-int
switch a80e0df8-16ca-466b-a34a-4d8df476b217 (incus-net5-ls-ext)
port incus-net5-ls-ext-lsp-router
type: router
router-port: incus-net5-lr-lrp-ext
port incus-net5-ls-ext-lsp-provider
type: localnet
addresses: ["unknown"]
router 5e072d2d-cac4-48eb-b1f8-bcf704b839a3 (incus-net5-lr)
port incus-net5-lr-lrp-int
mac: "10:66:6a:c7:e8:95"
networks: ["192.168.0.1/24"]
port incus-net5-lr-lrp-ext
mac: "10:66:6a:c7:e8:95"
networks: ["100.64.0.3/11"]
nat 30b0cae2-565f-4145-a23b-27400d0c64d2
external ip: "100.64.0.3"
logical ip: "192.168.0.0/24"
type: "snat"
username@incus-test:~$ sudo ovn-sbctl show
Chassis "30316039-f75a-4171-bbcd-2c9bd7b315c5"
hostname: incus-test
Encap geneve
ip: "127.0.0.1"
options: {csum="true"}
Port_Binding cr-incus-net5-lr-lrp-ext
Port_Binding incus-net5-instance-639da7f5-e0a2-47e1-8e43-b016d8450378-eth0
username@incus-test:~$ sudo ovn-nbctl list load_balancer
_uuid : 6dc2d54c-86a3-4b32-8cac-ec5c69b16f40
external_ids : {}
health_check : []
ip_port_mappings : {}
name : incus-net5-lb-100.96.0.10-tcp
options : {}
protocol : tcp
selection_fields : []
vips : {"100.96.0.10:1024"="192.168.0.2:80"}
The ARP reply bit is missing here (As described here)
username@incus-test:~$ sudo ovn-sbctl list logical_flow | grep 100.96.0.10
match : "ct.new && ip4.dst == 100.96.0.10 && tcp.dst == 1024"
actions : "reg1 = 100.96.0.10; reg2[0..15] = 1024; ct_lb_mark;"
match : "reg0[2] == 1 && ip4.dst == 100.96.0.10 && tcp.dst == 1024"
match : "ip && ip4.dst == 100.96.0.10"
match : "reg7 == 0 && ip4.dst == 100.96.0.10/32"
match : "ct.new && !ct.rel && ip4 && ip4.dst == 100.96.0.10 && tcp && tcp.dst == 1024 && is_chassis_resident(\"cr-incus-net5-lr-lrp-ext\")"
Can you help me figure out if it’s my setup or if it’s really an OVN regression? Thanks!