Encounter a problem when check ovn network connectivity

chenchen · August 17, 2022, 9:51am

my ovn network (which name is net-PfxB5iyvs2) use physical network (which name is UPLINK) as a parent network, and containers (cont-lp8vzw3x05 & cont-UsxxJT3V87) are connecting to ovn network net-PfxB5iyvs2.

i can ping by ip from container cont-lp8vzw3x05 to container cont-UsxxJT3V87.
but i can’t ping by ip from container cont-lp8vzw3x05 to ip 8.8.8.8.

this is UPLINK:

root@lxdserver1:~# lxc network show UPLINK
config:
  dns.nameservers: 8.8.8.8
  ipv4.gateway: 172.31.30.1/24
  ipv4.ovn.ranges: 172.31.30.148-172.31.30.158
  volatile.last_state.created: "false"
description: ""
name: UPLINK
type: physical
used_by:
- /1.0/networks/net-PfxB5iyvs2
managed: true
status: Created
locations:
- lxdserver1
- lxdserver2
- lxdserver3

this is net-PfxB5iyvs2

root@lxdserver1:~# lxc network show net-PfxB5iyvs2 
config:
  bridge.mtu: "1442"
  ipv4.address: 10.208.226.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:5f60:f26c:14b1::1/64
  ipv6.nat: "true"
  network: UPLINK
  volatile.network.ipv4.address: 172.31.30.148
description: ""
name: net-PfxB5iyvs2
type: ovn
used_by:
- /1.0/instances/cont-UsxxJT3V87
- /1.0/instances/cont-lp8vzw3x05
managed: true
status: Created
locations:
- lxdserver2
- lxdserver3
- lxdserver1

these are all my instances:

root@lxdserver1:~# lxc ls
+-----------------+---------+-----------------------+-------------------------------------------------+-----------------+-----------+------------+
|      NAME       |  STATE  |         IPV4          |                      IPV6                       |      TYPE       | SNAPSHOTS |  LOCATION  |
+-----------------+---------+-----------------------+-------------------------------------------------+-----------------+-----------+------------+
| cont-UsxxJT3V87 | RUNNING | 10.208.226.3 (eth0)   | fd42:5f60:f26c:14b1:216:3eff:fe25:b797 (eth0)   | CONTAINER       | 0         | lxdserver1 |
+-----------------+---------+-----------------------+-------------------------------------------------+-----------------+-----------+------------+
| cont-lp8vzw3x05 | RUNNING | 10.208.226.2 (eth0)   | fd42:5f60:f26c:14b1:216:3eff:fe80:ff78 (eth0)   | CONTAINER       | 0         | lxdserver1 |
+-----------------+---------+-----------------------+-------------------------------------------------+-----------------+-----------+------------+
| cont-zbvKpXAE04 | RUNNING | 10.224.16.2 (eth0)    | fd42:8489:65e2:537f:216:3eff:fea0:d481 (eth0)   | CONTAINER       | 0         | lxdserver2 |
+-----------------+---------+-----------------------+-------------------------------------------------+-----------------+-----------+------------+
| vm-hwzMovi9m5   | RUNNING | 10.224.16.20 (enp5s0) | fd42:8489:65e2:537f:216:3eff:fe44:b948 (enp5s0) | VIRTUAL-MACHINE | 0         | lxdserver3 |
+-----------------+---------+-----------------------+-------------------------------------------------+-----------------+-----------+------------+

ping result:

root@lxdserver1:~# lxc exec cont-lp8vzw3x05 bash
[root@cont-lp8vzw3x05 ~]# ping 10.208.226.3
PING 10.208.226.3 (10.208.226.3) 56(84) bytes of data.
64 bytes from 10.208.226.3: icmp_seq=1 ttl=64 time=1.51 ms
64 bytes from 10.208.226.3: icmp_seq=2 ttl=64 time=0.079 ms
64 bytes from 10.208.226.3: icmp_seq=3 ttl=64 time=0.071 ms
^C
--- 10.208.226.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2008ms
rtt min/avg/max/mdev = 0.071/0.555/1.517/0.680 ms
[root@cont-lp8vzw3x05 ~]# ping 8.8.8.8     
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

ip a from host

root@lxdserver1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp11s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:07:3e:9c:72:77 brd ff:ff:ff:ff:ff:ff
    inet 172.31.30.129/24 brd 172.31.30.255 scope global enp11s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::207:3eff:fe9c:7277/64 scope link 
       valid_lft forever preferred_lft forever
3: enp11s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 00:07:3e:9c:72:78 brd ff:ff:ff:ff:ff:ff
4: enp6s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether f8:f2:1e:91:27:14 brd ff:ff:ff:ff:ff:ff
5: enp6s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether f8:f2:1e:91:27:15 brd ff:ff:ff:ff:ff:ff
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 12:1f:06:8c:6e:18 brd ff:ff:ff:ff:ff:ff
7: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether da:ac:92:d0:f9:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2c0b:baff:fef3:3fc8/64 scope link 
       valid_lft forever preferred_lft forever
8: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN group default qlen 1000
    link/ether 9a:6e:dc:ba:bd:96 brd ff:ff:ff:ff:ff:ff
10: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:e2:a4:41 brd ff:ff:ff:ff:ff:ff
    inet 10.224.16.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:8489:65e2:537f::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fee2:a441/64 scope link 
       valid_lft forever preferred_lft forever
40: lxdovn2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:07:3e:9c:72:78 brd ff:ff:ff:ff:ff:ff
48: veth1eac7213@if47: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether d2:6c:6c:6b:a7:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0
50: veth53019074@if49: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 46:fc:8e:07:89:9a brd ff:ff:ff:ff:ff:ff link-netnsid 2

ip r from host

root@lxdserver1:~#  ip r
default via 172.31.30.1 dev enp11s0f0 proto static 
10.224.16.0/24 dev lxdbr0 proto kernel scope link src 10.224.16.1 linkdown 
172.31.30.0/24 dev enp11s0f0 proto kernel scope link src 172.31.30.129

ip a from container cont-lp8vzw3x05

root@lxdserver1:~# lxc exec cont-lp8vzw3x05 bash
[root@cont-lp8vzw3x05 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
47: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:80:ff:78 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.208.226.2/24 brd 10.208.226.255 scope global dynamic eth0
       valid_lft 2181sec preferred_lft 2181sec
    inet6 fd42:5f60:f26c:14b1:216:3eff:fe80:ff78/64 scope global mngtmpaddr dynamic 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe80:ff78/64 scope link 
       valid_lft forever preferred_lft forever

ip r from container cont-lp8vzw3x05

[root@cont-lp8vzw3x05 ~]# ip r
default via 10.208.226.1 dev eth0 
10.208.226.0/24 dev eth0 proto kernel scope link src 10.208.226.2 
169.254.0.0/16 dev eth0 scope link metric 1047

I can’t connect to the external network through the ovn network.
If my instance is connected to lxdbr0, it can access the external network.
What is wrong with my ovn network configuration?

tomp · August 17, 2022, 2:59pm

Your output of lxc network show UPLINK was cut off.

Are all containers on the same host?

Please show ip a and ip r from host and inside the container.

Please also show lxc network show net-PfxB5iyvs2

chenchen · August 18, 2022, 2:32am

I have adjusted the style of my code block, Please have a look again. If the code block is too long, there should be a scroll bar.

I have a 3 node lxd cluster, Usually, my containers are not on the same host.
However, in the case I showed, cont-lp8vzw3x05 and cont-UsxxJT3V87 are on the same host.

chenchen · August 18, 2022, 2:38am

Here is my command to create UPLINK.

lxc network create UPLINK --type=physical parent=enp11s0f1 --target=lxdserver1
lxc network create UPLINK --type=physical parent=enp11s0f1 --target=lxdserver2
lxc network create UPLINK --type=physical parent=enp11s0f1 --target=lxdserver3
lxc network create UPLINK --type=physical ipv4.ovn.ranges=172.31.30.148-172.31.30.158 ipv4.gateway=172.31.30.1/24 dns.nameservers=8.8.8.8

tomp · August 18, 2022, 7:31am

Can any of the containers ping 8.8.8.8?

tomp · August 18, 2022, 7:31am

What does lxc network info net-PfxB5iyvs2 output?

chenchen · August 18, 2022, 7:55am

root@lxdserver1:~# lxc network show net-PfxB5iyvs2
config:
bridge.mtu: “1442”
ipv4.address: 10.208.226.1/24
ipv4.nat: “true”
ipv6.address: fd42:5f60:f26c:14b1::1/64
ipv6.nat: “true”
network: UPLINK
volatile.network.ipv4.address: 172.31.30.148
description: “”
name: net-PfxB5iyvs2
type: ovn
used_by:

/1.0/instances/cont-UsxxJT3V87
/1.0/instances/cont-lp8vzw3x05
managed: true
status: Created
locations:
lxdserver2
lxdserver3
lxdserver1

chenchen · August 18, 2022, 7:56am

none of them could ping 8.8.8.8.

tomp · August 18, 2022, 8:01am

OK lets start with some basic network diagnostics:

What is the output of lxc network info net-PfxB5iyvs2?
Can you ping the uplink gateway 172.31.30.1 from the instances?
Can you ping the OVN network’s external virtual router address (volatile.network.ipv4.address) 172.31.30.148 from the gateway (or another host in the same network)?
When you ping 8.8.8.8 from the instances do you see any traffic on the physical uplink interface enp11s0f1 on any of the LXD cluster members (using sudo tcpdump -i enp11s0f1 -nn)?

chenchen · August 26, 2022, 7:11am

My English is not good. At first, I didn’t understand you correctly, But now I understand. I didn‘t realize that i need to tcpdump 3 physical uplink interface(all of them named enp11s0f1) at first.
Your last point of view solved my problem. I just found out now, you have already told me the answer. Thank you very very much.

tomp · August 26, 2022, 7:15am

What was the issue in the end?

chenchen · August 29, 2022, 3:20am

my enp11s0f1 is faulty on lxdserver2, enp11s0f1 on lxdserver1 is ok, enp11s0f1 on lxdserver 3 is ok too. It’s a hardware problem, I’ve already fixed it.

I drew a network topology to explain my understanding of ovn networks,
Please check whether my understanding of the underlying implementation of ovn network is correct?

tomp · August 30, 2022, 7:52am

Yes so in this case the “localnet” is the uplink network, the top “switch” is the OVS switch that LXD uses to connect the uplink network to and the “router” and 2nd “switch” are the OVN virtual entities created for each LXD OVN network.

If you created multiple LXD OVN networks connected to the same uplink network then each would have its own virtual router and switch connected to the top OVS switch (which is connected to the uplink network).

For OVN networks in LXD clusters, the virtual router and switch exist on all cluster members at the same time, but the external port on the router connected to the uplink network is only ever active on one LXD cluster member at once (you can see this via lxc network info <ovn_network>)