I have the following network setup (IPs redacted as they are public IPs):
xxxx:xxxx:8a::/44 is assigned to LXD host
xxxx:xxxx:8f::/44 is assigned to the gateway (default gateway for ipv6 on lxd host)
The LXD host can access IPv6 sites fine
lxdbr0 has xxxx:xxxx:8a::1/48 assigned. I currently have nat enabled for ipv6 (more on that later)
In the current setup, after LXD and the containers start up, everything works fine. However, the ipv6 IPs (in ip -6 a) and routes (in ip -6 r) have an expiry timer. Once these expire, they do not come back / renew, and IPv6 connectivity completely breaks as a result.
With NAT off (who even uses NAT with IPv6 ), the same issue persists, however routing is also broken. I would expect it to route to 8a::1 and then to 8f via 8a, however trace route shows it gets to the gateway (8a::1) and then nothing more. I enabled IPv6 forwarding on the host in sysctl, but this did not help.
Your behavior sounds like you may have some firewalling messing with things.
Do you perhaps have Docker installed on your host, or are using ufw or firewalld? All three of those have a bit of a tendency to mess with firewalling in interesting ways which could cause that.
I can recreate this on a completely stock install of lxd and ubuntu server 20.04. Weirdly, if I have just changed the IPv6 address for the lxd bridge, then it will renew several times as expected, but then stop renewing altogether afterwards.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:14:0b:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 xxxx:xxxx:8c:1:216:3eff:fe14:ba4/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3591sec preferred_lft 3591sec
inet6 fe80::216:3eff:fe14:ba4/64 scope link
valid_lft forever preferred_lft forever
xxxx:xxxx:8c:1::/64 dev eth0 proto ra metric 100 expires 3531sec pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::216:3eff:fe7f:94da dev eth0 proto ra metric 100 expires 1731sec mtu 1500 pref medium
Host:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 56:8d:b1:6b:09:ed brd ff:ff:ff:ff:ff:ff
inet xxx.xxx.xxx.xxx/29 brd xxx.xxx.xxx.xxx scope global ens18
valid_lft forever preferred_lft forever
inet6 fe80::548d:b1ff:fe6b:9ed/64 scope link
valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 66:b7:70:f4:80:be brd ff:ff:ff:ff:ff:ff
inet 10.0.0.100/24 brd 10.0.0.255 scope global ens19
valid_lft forever preferred_lft forever
inet6 xxxx:xxxx:8c::/44 scope global
valid_lft forever preferred_lft forever
inet6 fe80::64b7:70ff:fef4:80be/64 scope link
valid_lft forever preferred_lft forever
4: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:25:1e:31 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 xxxx:xxxx:8c:1::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe25:1e31/64 scope link
valid_lft forever preferred_lft forever
::1 dev lo proto kernel metric 256 pref medium
xxxx:xxxx:8c:1::/64 dev lxdbr0 proto kernel metric 256 linkdown pref medium
xxxx:xxxx:80::/44 dev ens19 proto kernel metric 256 pref medium
fe80::/64 dev ens19 proto kernel metric 256 pref medium
fe80::/64 dev ens18 proto kernel metric 256 pref medium
fe80::/64 dev lxdbr0 proto kernel metric 256 linkdown pref medium
default via xxxx:xxxx:8f:: dev ens19 proto static metric 1024 pref medium
(i switched the ipv6 block to a /64 to see if it would help, however this had no effect)
Thanks, also please show output of ps aux | grep dnsmasq.
Also, please can you setup a ping from the container out to an IPv6 destination beyond the host, and then run a sudo tcpdump -i lxdbr0 -nn host <target_address> on the host, and also sudo tcpdump -i <external interface> -nn host <target_address> to try and see where the routing breakage is occurring.
After tcpdumping my interfaces, I discovered an issue with my upstream router’s route table, which has been corrected. The container can now access the internet without nat fine.
ip6tables-save:
# Generated by ip6tables-save v1.8.4 on Tue May 10 10:57:52 2022
*filter
:INPUT ACCEPT [1168:131064]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [214:19870]
:lxd_nic_lxdbr0 - [0:0]
-A INPUT -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j lxd_nic_lxdbr0
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 136 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 133 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 4 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 3 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 2 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 1 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 547 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j lxd_nic_lxdbr0
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 136 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 128 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 4 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 3 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 2 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p ipv6-icmp -m icmp6 --icmpv6-type 1 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 547 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
COMMIT
# Completed on Tue May 10 10:57:52 2022
# Generated by ip6tables-save v1.8.4 on Tue May 10 10:57:52 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Tue May 10 10:57:52 2022
# Generated by ip6tables-save v1.8.4 on Tue May 10 10:57:52 2022
*mangle
:PREROUTING ACCEPT [3835:398743]
:INPUT ACCEPT [1868:234055]
:FORWARD ACCEPT [1125:103944]
:OUTPUT ACCEPT [864:86071]
:POSTROUTING ACCEPT [2012:192911]
COMMIT
# Completed on Tue May 10 10:57:52 2022
# Generated by ip6tables-save v1.8.4 on Tue May 10 10:57:52 2022
*raw
:PREROUTING ACCEPT [3835:398743]
:OUTPUT ACCEPT [864:86071]
COMMIT
# Completed on Tue May 10 10:57:52 2022