IPv6 with Containers -> A reboot results in a dadfailed in address assignment from dhcp (OVN)

Incus
1

Hey there!

I ran into a somewhat strange situation: When I reboot a LXC container, that has an IPv6 address from OVN via SLAAC, the container shows on eth0 after the reboot “dadfailed” meaning that it could not occupie the old address again, because it is still in use.

You can fix this by issuing:

sysctl -w net.ipv6.conf.eth0.accept_dad=0
ip -6 addr flush dev eth0 scope global
systemctl restart systemd-networkd

So I thought the reboot was “too quick” for OVN to notice, but even a hard shutdown, waiting a minute and starting again, and the issue persists.

I am not sure though if this is an Incus issue or one with OVN.
Has anyone experienced similar behaviour?

2

May be worth running some tcpdump from within the container to see what’s responding to the NDP query.

3

Thanks for the hint, I was able to dig a little deeper and found something. Even though the network has set “ipv6.nat” to false, it still creates a “dnat_and_snat” entry in OVN, which is I think the reason for the DAD reply:

# grep for the mac of one of the affected containers
root@apricot:~# ovn-nbctl list NAT | grep -B5 -A10 feb5:f75f

_uuid               : e7bf5107-f2b7-46e3-ade3-cfa0da0eb7c5
allowed_ext_ips     : []
exempted_ext_ips    : []
external_ids        : {}
external_ip         : "2001:XXXX:XXXX:600:1266:6aff:feb5:f75f"
external_mac        : []
external_port_range : ""
gateway_port        : []
logical_ip          : "2001:XXXX:XXXX:600:1266:6aff:feb5:f75f"
logical_port        : []
options             : {stateless="true"}
type                : dnat_and_snat
root@apricot:~# incus network show public --project office
config:
  bridge.mtu: "1422"
  dns.nameservers: 2001:4860:4860::8888,8.8.8.8
  ipv4.address: 10.27.186.1/24
  ipv4.nat: "true"
  ipv6.address: 2001:XXXX:XXXX:600::1/64
  ipv6.nat: "false"
  network: UPLINK
  volatile.network.ipv4.address: 192.168.33.26
  volatile.network.ipv6.address: 2001:XXXX:XXXX:0:1266:6aff:fe0c:7915
description: Public network
name: public
type: ovn
used_by:
  - /1.0/instances/applications-controlplane?project=office
  - /1.0/instances/continuwuity?project=office
managed: true
status: Created
locations:
  - apricot
  - banana
  - cherry
project: office

On a container reboot the Linux network stack in the container get’s a reply from the host system on the DAD request:

root@banana:~# tcpdump -eni veth49af179f icmp6
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on veth49af179f, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:39:52.897996 10:66:6a:0c:79:15 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 158: fe80::1266:6aff:fe0c:7915 > ff02::1: ICMP6, router advertisement, length 104
03:40:09.304265 10:66:6a:c2:eb:7c > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: fe80::1266:6aff:fec2:eb7c > ff02::2: ICMP6, router solicitation, length 16
03:40:09.305864 10:66:6a:0c:79:15 > 10:66:6a:c2:eb:7c, ethertype IPv6 (0x86dd), length 158: fe80::1266:6aff:fe0c:7915 > fe80::1266:6aff:fec2:eb7c: ICMP6, router advertisement, length 104 03:40:09.342361 10:66:6a:c2:eb:7c > 33:33:ff:c2:eb:7c, ethertype IPv6 (0x86dd), length 86: :: > ff02::1:ffc2:eb7c: ICMP6, neighbor solicitation, who has 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c, length 32
03:40:09.343584 10:66:6a:0c:79:15 > 10:66:6a:c2:eb:7c, ethertype IPv6 (0x86dd), length 86: 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c > ::: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c, length 32
03:40:09.343629 10:66:6a:c2:eb:7c > 33:33:ff:c2:eb:7c, ethertype IPv6 (0x86dd), length 86: fe80::1266:6aff:fec2:eb7c > ff02::1:ffc2:eb7c: ICMP6, neighbor solicitation, who has 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c, length 32
03:40:09.344404 10:66:6a:0c:79:15 > 10:66:6a:c2:eb:7c, ethertype IPv6 (0x86dd), length 86: 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c > fe80::1266:6aff:fec2:eb7c: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c, length 32
03:40:10.371594 10:66:6a:c2:eb:7c > 33:33:ff:c2:eb:7c, ethertype IPv6 (0x86dd), length 86: fe80::1266:6aff:fec2:eb7c > ff02::1:ffc2:eb7c: ICMP6, neighbor solicitation, who has 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c, length 32
03:40:10.373018 10:66:6a:0c:79:15 > 10:66:6a:c2:eb:7c, ethertype IPv6 (0x86dd), length 86: 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c > fe80::1266:6aff:fec2:eb7c: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c, length 32
03:40:11.395652 10:66:6a:c2:eb:7c > 33:33:ff:c2:eb:7c, ethertype IPv6 (0x86dd), length 86: fe80::1266:6aff:fec2:eb7c > ff02::1:ffc2:eb7c: ICMP6, neighbor solicitation, who has 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c, length 32
03:40:11.397081 10:66:6a:0c:79:15 > 10:66:6a:c2:eb:7c, ethertype IPv6 (0x86dd), length 86: 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c > fe80::1266:6aff:fec2:eb7c: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:600:1266:6aff:fec2:eb7c, length 32
^C
11 packets captured
11 packets received by filter
0 packets dropped by kernel
root@banana:~#

I also took a look at the router:

root@apricot:~# ovn-nbctl show | grep -A20 -B5 incus-net11
(...)
router 2ca929ab-c250-4cf7-b92d-cfb768270b51 (

incus-net11

-lr)
    port

incus-net11

-lr-lrp-int
        mac: "10:66:6a:0c:79:15"
        networks: ["10.27.186.1/24", "2001:XXXX:XXXX:600::1/64"]
    port

incus-net11

-lr-lrp-ext
        mac: "10:66:6a:0c:79:15"
        networks: ["192.168.33.26/24", "2001:XXXX:XXXX:0:1266:6aff:fe0c:7915/64"]
    nat 794fd29b-c4fe-4137-9f9d-a81e8d0602af
        external ip: "192.168.33.26"
        logical ip: "10.27.186.0/24"
        type: "snat"
    nat e7bf5107-f2b7-46e3-ade3-cfa0da0eb7c5
        external ip: "2001:XXXX:XXXX:600:1266:6aff:feb5:f75f"
        logical ip: "2001:XXXX:XXXX:600:1266:6aff:feb5:f75f"
        type: "dnat_and_snat"
    nat ef4330ac-e3fd-4a76-96fe-ee7f64aecfa1
        external ip: "2001:XXXX:XXXX:600:1266:6aff:fe82:f809"
        logical ip: "2001:XXXX:XXXX:600:1266:6aff:fe82:f809"
        type: "dnat_and_snat"

So I manually deleted the NAT rule:
ovn-nbctl remove Logical_Router incus-net11-lr nat e7bf5107-f2b7-46e3-ade3-cfa0da0eb7c5

and restarted the container. To my surprise, the nat rule just came back with a new UUID.
This looks like a bug, right?

Edit: I am on incus 7.3:
root@apricot:~# incus version
Client version: 7.3
Server version: 7.3

4

Hmm, yeah, that does look like a bug. I’ll take a look…

5

I did some digging.

So what’s going on here is that the default ovn.ingress_mode is l2proxy. Meaning that when NAT isn’t in use, OVN will setup a NAT rule like the one you’re seeing to cause its own router to respond to NDP on the uplink network.

If you are in an environment where you have proper routing, that is, you have a route for your IPv6 subnet going to the correct OVN virtual router uplink address, then you don’t need the l2proxy and can use routed instead which will eliminate the NAT rule trick.

That’s why I’m not seeing it in any of my production environments as they all use routed mode with BGP advertisements to handle the correct next-hop into the OVN router.

6

Newer OVN appears to have a native arp_proxy mode that handles IPv6 but our minimum supported version of OVN does not have that feature, hence why we’re still using the NAT workaround.

I’m now preparing a branch which will bump our OVS and OVN baseline versions a fair bit which will then allow switching the logic to the upstream arp_proxy option.

Note that the behavior you’re seeing still feels like an OVN bug as OVN’s generated flow rule really shouldn’t match traffic coming from its target.

7

Agreed, looks like an OVN problem to me, too. Unfortunately, I still haven’t managed to get my setup working with BGP. But I am in a lot of edge cases anyway because I run my cluster from a consumer internet connection, tunnel a /48 over a dynamic public IPv4 to get static IPs, etc. I even found a hardware bug in my Mikrotik Router with the IPv6 datapth. The only solution is to permanently torch the /48 VLAN so the traffic is forced through the Routers CPU :rofl:

8

Unless you very frequently create OVN nertworks, you can use the routed mode by looking at incus network info NAME and putting a static route for that network’s IPv6 subnet through the OVN router’s IPv6 address you get in the network info.