LXD 4.24 IPv6 renewal and routing issues

[natekomodo]

I have the following network setup (IPs redacted as they are public IPs):

xxxx:xxxx:8a::/44 is assigned to LXD host
xxxx:xxxx:8f::/44 is assigned to the gateway (default gateway for ipv6 on lxd host)
The LXD host can access IPv6 sites fine
lxdbr0 has xxxx:xxxx:8a::1/48 assigned. I currently have nat enabled for ipv6 (more on that later)

In the current setup, after LXD and the containers start up, everything works fine. However, the ipv6 IPs (in ip -6 a) and routes (in ip -6 r) have an expiry timer. Once these expire, they do not come back / renew, and IPv6 connectivity completely breaks as a result.

With NAT off (who even uses NAT with IPv6 ), the same issue persists, however routing is also broken. I would expect it to route to 8a::1 and then to 8f via 8a, however trace route shows it gets to the gateway (8a::1) and then nothing more. I enabled IPv6 forwarding on the host in sysctl, but this did not help.

Any ideas?

[stgraber (Stéphane Graber)]

Your behavior sounds like you may have some firewalling messing with things.

Do you perhaps have Docker installed on your host, or are using ufw or firewalld? All three of those have a bit of a tendency to mess with firewalling in interesting ways which could cause that.

[natekomodo]

I dont have docker or firewalld. I do have ufw but i have not touched it since install (ubuntu 20.04 server)

[natekomodo]

I can recreate this on a completely stock install of lxd and ubuntu server 20.04. Weirdly, if I have just changed the IPv6 address for the lxd bridge, then it will renew several times as expected, but then stop renewing altogether afterwards.

[tomp (Thomas Parrott)]

Please can you show ip a and ip r on the host and inside the container. Thanks

[natekomodo]

In container prior to address expiration:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:14:0b:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 xxxx:xxxx:8c:1:216:3eff:fe14:ba4/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 3591sec preferred_lft 3591sec
    inet6 fe80::216:3eff:fe14:ba4/64 scope link
       valid_lft forever preferred_lft forever

xxxx:xxxx:8c:1::/64 dev eth0 proto ra metric 100 expires 3531sec pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::216:3eff:fe7f:94da dev eth0 proto ra metric 100 expires 1731sec mtu 1500 pref medium

Host:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 56:8d:b1:6b:09:ed brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxx/29 brd xxx.xxx.xxx.xxx scope global ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::548d:b1ff:fe6b:9ed/64 scope link
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 66:b7:70:f4:80:be brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.100/24 brd 10.0.0.255 scope global ens19
       valid_lft forever preferred_lft forever
    inet6 xxxx:xxxx:8c::/44 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::64b7:70ff:fef4:80be/64 scope link
       valid_lft forever preferred_lft forever
4: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:25:1e:31 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 xxxx:xxxx:8c:1::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe25:1e31/64 scope link
       valid_lft forever preferred_lft forever

::1 dev lo proto kernel metric 256 pref medium
xxxx:xxxx:8c:1::/64 dev lxdbr0 proto kernel metric 256 linkdown pref medium
xxxx:xxxx:80::/44 dev ens19 proto kernel metric 256 pref medium
fe80::/64 dev ens19 proto kernel metric 256 pref medium
fe80::/64 dev ens18 proto kernel metric 256 pref medium
fe80::/64 dev lxdbr0 proto kernel metric 256 linkdown pref medium
default via xxxx:xxxx:8f:: dev ens19 proto static metric 1024 pref medium

(i switched the ipv6 block to a /64 to see if it would help, however this had no effect)

[tomp (Thomas Parrott)]

Can you show lxc network show lxdbr0 please?

[natekomodo]

config:
  ipv4.address: none
  ipv6.address: xxxx:xxxx:8c:1::1/64
  ipv6.nat: "false" # setting to true breaks routing and the renewal issue persists
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/test
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

[tomp (Thomas Parrott)]

Thanks, also please show output of ps aux | grep dnsmasq.

Also, please can you setup a ping from the container out to an IPv6 destination beyond the host, and then run a sudo tcpdump -i lxdbr0 -nn host <target_address> on the host, and also sudo tcpdump -i <external interface> -nn host <target_address> to try and see where the routing breakage is occurring.

[tomp (Thomas Parrott)]

@natekomodo

I can recreate this on a completely stock install of lxd and ubuntu server 20.04. Weirdly, if I have just changed the IPv6 address for the lxd bridge, then it will renew several times as expected, but then stop renewing altogether afterwards.

Please also show output of sudo ip6tables-save and sudo nft list ruleset (if available)

[natekomodo]

Heres dnsmasq:

lxd         7847  0.0  0.0   7200  3560 ?        Ss   09:57   0:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --dhcp-rapid-commit --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=xxxx:xxxx:8c:1::1 --enable-ra --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range ::,constructor:lxdbr0,ra-stateless,ra-names -s lxd --interface-name _gateway.lxd,lxdbr0 -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd

After tcpdumping my interfaces, I discovered an issue with my upstream router’s route table, which has been corrected. The container can now access the internet without nat fine.

ip6tables-save:

# Generated by ip6tables-save v1.8.4 on Tue May 10 10:57:52 2022
*filter
:INPUT ACCEPT [1168:131064]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [214:19870]
:lxd_nic_lxdbr0 - [0:0]
-A INPUT -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j lxd_nic_lxdbr0
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 136 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 133 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 4 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 3 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 2 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 1 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 547 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j lxd_nic_lxdbr0
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 136 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 128 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 4 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 3 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 2 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 1 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 547 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
COMMIT
# Completed on Tue May 10 10:57:52 2022
# Generated by ip6tables-save v1.8.4 on Tue May 10 10:57:52 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Tue May 10 10:57:52 2022
# Generated by ip6tables-save v1.8.4 on Tue May 10 10:57:52 2022
*mangle
:PREROUTING ACCEPT [3835:398743]
:INPUT ACCEPT [1868:234055]
:FORWARD ACCEPT [1125:103944]
:OUTPUT ACCEPT [864:86071]
:POSTROUTING ACCEPT [2012:192911]
COMMIT
# Completed on Tue May 10 10:57:52 2022
# Generated by ip6tables-save v1.8.4 on Tue May 10 10:57:52 2022
*raw
:PREROUTING ACCEPT [3835:398743]
:OUTPUT ACCEPT [864:86071]
COMMIT
# Completed on Tue May 10 10:57:52 2022

nft list ruleset returns nothing.

[tomp (Thomas Parrott)]

Great!

We’ve seen something similar with dnsmasq in the past, please can you try this and see if it fixes it for you:

Random IPv6 address dropouts in lxdbr0

OK lets set that option to false then. Try seeing changing the interval helps: lxc network set lxdbr0 raw.dnsmasq="ra-param=lxdbr0,10" You should then see the RAs every 10s and at least inside my Focal container I can see the lifetime increase back to 86400secs every 10s for the IPv6 address: inet6 fd42:5433:626e:4c0b:216:3eff:fe35:1f9d/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 86400sec preferred_lft 86400sec