OVN Container Cannot Access Internet via Uplink or Bridge

Incus
, ,
1

Hello! Over the past week, I started learning about Incus clustering and OVN. After some trial and error with the setup (still learning), I successfully got it working across three cluster nodes with multiple virtual machines.

Initially, I created an OVN network directly on the physical interface, and it worked great. However, I later decided to use a bridge-based uplink so that both containers and VMs could access the internet.

I created the bridge network using the following commands:

$ incus network create incus-bridge --type=bridge --target cluster-node-01
Network incus-bridge pending on member cluster-node-01
$ incus network create incus-bridge --type=bridge --target cluster-node-02
Network incus-bridge pending on member cluster-node-02
$ incus network create incus-bridge --type=bridge --target cluster-node-03
Network incus-bridge pending on member cluster-node-03
$ incus network create incus-bridge --type=bridge
Network incus-bridge created

Then, I set it up as the uplink:

$ incus network create UPLINK --type=physical parent=incus-bridge --target=cluster-node-01
$ incus network create UPLINK --type=physical parent=incus-bridge --target=cluster-node-02
$ incus network create UPLINK --type=physical parent=incus-bridge --target=cluster-node-03
$ incus network create UPLINK --type=physical \
   ipv4.ovn.ranges=10.12.10.70-10.12.10.72 \
   ipv4.gateway=10.12.10.1/24 \
   dns.nameservers=192.168.50.253,8.8.8.8

At this point, everything works correctly. Any VM or container using UPLINK or incus-bridge gets internet and DNS access. :+1:

However, the issue arises after creating an OVN network:

text

$ incus network create cluster-default-network --type=ovn

As you can see, it automatically uses UPLINK as the parent network. Containers and VMs on this OVN network get IP addresses and can communicate with each other across cluster nodes. But no matter what I try, they cannot access DNS or the public internet through the bridge or uplink.

What am I missing or doing wrong?

2

You’ll want to look at incus network info cluster-default-network.
That should tell you what server is the current virtual router for it and what the external IP addresses are.

You’ll then want to go on that server and see if you can ping that address.
If you can, then the ingress into OVN is likely fine, at which point you’d want to break out tcpdump to dump the traffic on incus-bridge on that system to see what’s going on.

1 Like
3

Just tried that and cannot ping it from inside of container, I gave the UPLINK same address used by Incus nodes and ovn ranges

$ incus network info cluster-default-network
Name: cluster-default-network
MAC address: 10:66:6a:42:f2:bd
MTU: 1442
State: up
Type: broadcast

IP addresses:
  inet  10.205.172.1/24 (link)
  inet6 fd42:58dd:d167:7a00::1/64 (link)

OVN:
  Chassis: ubuntu2204
  Logical router: incus-net25-lr
  Logical switch: incus-net25-ls-int
  IPv4 uplink address: 10.12.10.70
$ ping 10.12.10.70
PING 10.12.10.70 (10.12.10.70) 56(84) bytes of data.
64 bytes from 10.12.10.70: icmp_seq=1 ttl=64 time=0.065 ms
64 bytes from 10.12.10.70: icmp_seq=2 ttl=64 time=0.045 ms
^C
--- 10.12.10.70 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1052ms
rtt min/avg/max/mdev = 0.045/0.055/0.065/0.010 ms
$ incus exec rockylinux-on-01 -- ping 10.12.10.70
PING 10.12.10.70 (10.12.10.70) 56(84) bytes of data.
^C
--- 10.12.10.70 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5154ms

$ incus exec rockylinux-on-02 -- ping 10.12.10.70
PING 10.12.10.70 (10.12.10.70) 56(84) bytes of data.
^C
--- 10.12.10.70 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5111ms
4

Can the containers ping 10.205.172.1?

5

Yes, they can ping the gateway of cluster-default-network,

just used tcpdump on incus-bridge and didn’t saw any traffic reach it from cluster-default-network

6

after some reconfiguration of network now I got response from tcpdump :slight_smile:

$ tcpdump -i incus-bridge
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on incus-bridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:47:36.469212 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:37.510673 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:38.533494 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:39.557927 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:40.581765 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:41.606407 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:42.629939 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:43.653917 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:44.677570 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:45.701573 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:46.725906 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:47.749679 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:48.773492 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:49.798110 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:50.822170 ARP, Request who-has _gateway tell ubuntu2204, length 28
12:47:51.845809 ARP, Request who-has _gateway tell ubuntu2204, length 28
7

Is it possible to be related duplicated issue like this one?

8

I’m assuming you already tried to restart things?
In situations like those, I’d probably give restarting both ovn-controller and incus a try, to basically force the uplink to get setup again.

Some of it clearly is working since you can ping the OVN router.

9

Just tried restart Incus, Incus.socket and ovn-controller and still facing same issue. also crrated containers in different nodes ad restart all services. :frowning:

I will try to downgrade OVN to test if it will make any different

10

managed to resolve the issue by removing the UPLINK interface and used incus-bridge directly

previous setup:

incus-bridge <---> UPLINK <---> cluster-default-network

the new setup:

incus-bridge <---> cluster-default-network

this is the manifest for incus-bridge:

project: default
name: incus-bridge
description: 'CAUTION: Do not modify or delete this interface. It is manually configured for OVN. Changing it may cause virtual machines to lose internet connectivity.'
type: bridge
config:
  ipv4.address: 10.6.29.1/16
  ipv4.dhcp: 'true'
  ipv4.dhcp.ranges: 10.6.32.1-10.6.32.254
  ipv4.nat: 'true'
  ipv4.ovn.ranges: 10.6.31.1-10.6.31.254
  ipv6.address: none

and the following is manifest for cluster-default-network:

project: default
name: cluster-default-network
description: 'CAUTION: Do not modify or delete this interface. It is manually configured for OVN. Changing it may cause virtual machines to lose internet connectivity.'
type: ovn
config:
  volatile.network.ipv4.address: 10.6.31.1
  bridge.mtu: '1442'
  ipv4.address: 10.6.30.1/24
  ipv4.nat: 'true'
  ipv6.address: none
  network: incus-bridge

I don’t have IPv6 subset on this setup and I’m planning to experiment with it later :+1: