LXD OVN HA CEPH internet problem

LXD
1

Hi

I have 3 nodes that is a lxd cluster with ovn in ha mode that @tomp described in OVN high availability cluster tutorial.

I can ping ovn virtual routers of each ovn networks in the node that is gateway.
but the problem is that containers doesnt have internet access.

I created networks like this:

ip link add dummybr0 type bridge # Create dummy uplink bridge.
ip address add 192.0.2.1/24 dev dummybr0
ip address add 2001:db8:1:1::1/64 dev dummybr0
ip link set dummybr0 up
lxc network create dummy --type=physical parent=dummybr0 --target=node1
lxc network create dummy --type=physical parent=dummybr0 --target=node2
lxc network create dummy --type=physical parent=dummybr0 --target=node3
lxc network create dummy --type=physical ipv4.gateway=192.0.2.1/24     ipv4.ovn.ranges=192.0.2.10-192.0.2.19 ipv4.routes=157.119.190.128/29    ovn.ingress_mode=routed
lxc network create ovn0 --type=ovn network=dummy
lxc launch images:ubuntu/20.04 c1 --network ovn0
lxc network get ovn0 volatile.network.ipv4.address => # i can ping this in node1 for example

lxc shell c1 => ping 8.8.8.8 doesnt work

But when i use lxdbr0 for containers they have internet :frowning:

2

You’ve created a dummybr0 using 192.0.2.1/24, this means that each OVN router will SNAT to 192.0.2.x.

They will send traffic to 192.0.2.1 and then its up to the active chassis to route the traffic externally.

Have you configured routing or SNAT appropriate to allow traffic emerging from dummybr0 using addresses in 192.0.2.0/24 to reach the internet?

3

My uplink is bridge interface br0. no I didnt wrote any route or snat.
In my container i can’t ping 192.0.2.1.

4

Your post says the uplink bridge is dummybr0

5

On the active chassis what do you see from sudo tcpdump -i dummybr0 -nn when pinging 192.0.2.1 from one of the containers?

6

Nothing.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on dummybr0, link-type EN10MB (Ethernet), capture size 262144 bytes
7

I’m sorry. the br0 bridge is connected to physical interface that has internet.

8

I’m so confused :slight_smile:

9

Please show:

lxc network show dummy --target=node1
lxc network show dummy --target=node2
lxc network show dummy --target=node3

Please also show:

lxc network show ovn0

and

lxc network info ovn0 --target=node1
lxc network info ovn0 --target=node2
lxc network info ovn0 --target=node3
10 
lxc network show dummy --target=node1
config:
  ipv4.gateway: 192.0.2.1/24
  ipv4.ovn.ranges: 192.0.2.10-192.0.2.19
  ipv4.routes: 157.119.x.x/29
  ovn.ingress_mode: routed
  parent: dummybr0
  volatile.last_state.created: "false"
description: ""
name: dummy
type: physical
used_by:
- /1.0/networks/ali0
- /1.0/networks/ovn1
managed: true
status: Created
locations:
- node1
- node2
- node3

lxc network show dummy --target=node2
config:
  ipv4.gateway: 192.0.2.1/24
  ipv4.ovn.ranges: 192.0.2.10-192.0.2.19
  ipv4.routes: 157.119.xx.xx/29
  ovn.ingress_mode: routed
  parent: dummybr0
  volatile.last_state.created: "false"
description: ""
name: dummy
type: physical
used_by:
- /1.0/networks/ali0
- /1.0/networks/ovn1
managed: true
status: Created
locations:
- node1
- node2
- node3

lxc network show dummy --target=node3
config:
  ipv4.gateway: 192.0.2.1/24
  ipv4.ovn.ranges: 192.0.2.10-192.0.2.19
  ipv4.routes: 157.119.xx.xx/29
  ovn.ingress_mode: routed
  parent: dummybr0
  volatile.last_state.created: "false"
description: ""
name: dummy
type: physical
used_by:
- /1.0/networks/ali0
- /1.0/networks/ovn1
managed: true
status: Created
locations:
- node3
- node1
- node2

lxc network show ovn1
config:
  bridge.mtu: "1442"
  ipv4.address: 10.247.233.1/24
  ipv4.nat: "true"

  ipv6.address: fd42:fbb2:cc6c:a49e::1/64
  ipv6.nat: "true"
  network: dummy
  volatile.network.ipv4.address: 192.0.2.11
description: ""
name: ovn1
type: ovn
used_by:
- /1.0/instances/c2
managed: true
status: Created
locations:
- node1
- node2
- node3
11

And the info commands too please.

12 
lxc network info ovn1 --target=node1
Name: ovn1
MAC address: 00:16:3e:65:bb:fa
MTU: 1442
State: up
Type: broadcast

IP addresses:
  inet	10.247.233.1/24 (link)
  inet6	fd42:fbb2:cc6c:a49e::1/64 (link)

Network usage:
  Bytes received: 0B
  Bytes sent: 0B
  Packets received: 0
  Packets sent: 0

OVN:
  Chassis: node2

lxc network info ovn1 --target=node2
Error: Failed to run: ovn-nbctl --timeout=10 --db tcp:10.0.0.50:6641,tcp:10.0.0.60:6641,tcp:10.0.0.70:6641 --wait=sb --no-headings --data=bare --format=csv --columns=mac find Logical_Router_Port name=lxd-net100-lr-lrp-ext: ovn-nbctl: tcp:10.0.0.50:6641,tcp:10.0.0.60:6641,tcp:10.0.0.70:6641: database connection failed ()

lxc network info ovn1 --target=node3
Error: Failed to run: ovn-nbctl --timeout=10 --db tcp:10.0.0.50:6641,tcp:10.0.0.60:6641,tcp:10.0.0.70:6641 --wait=sb --no-headings --data=bare --format=csv --columns=mac find Logical_Router_Port name=lxd-net100-lr-lrp-ext: ovn-nbctl: tcp:10.0.0.50:6641,tcp:10.0.0.60:6641,tcp:10.0.0.70:6641: database connection failed ()
13

OK so on node2 can you run that tcpdump command, and then go into a container that is also on node2 and ping 192.0.2.1 and show what tcpdump records.

14

That looks like a bug, please can you log at Issues · lxc/incus · GitHub

1 Like
15 
sudo tcpdump -i dummybr0 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on dummybr0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:43:58.859872 IP 157.119.x.x > 192.0.2.1: ICMP echo request, id 362, seq 19, length 64
13:43:59.883889 IP 157.119.x.x > 192.0.2.1: ICMP echo request, id 362, seq 20, length 64
13:44:00.907782 IP 157.119.x.x > 192.0.2.1: ICMP echo request, id 362, seq 21, length 64
13:44:01.931816 IP 157.119.x.x > 192.0.2.1: ICMP echo request, id 362, seq 22, length 64
13:44:02.955716 IP 157.119.x.x > 192.0.2.1: ICMP echo request, id 362, seq 23, length 64
16

OK great so there’s the requests. But they are coming from 157.119.x.x rather than the OVN router’s address. Do you have a container with 157.119.x.x addresses configured and an ipv4.routes.external configured on the NIC?

17

No
OVN router’s address is 192.0.2.10?

18

Where is this coming from?

19

it is set on br0.

20

So is that a ping from inside the OVN container?
Why is its source address in the 157.119.x.x range, what does ip a inside the container sending the ping show?