Hi all, I’m struggling to get direct (non-NAT) IPv6 working inside LXC with veth.
The summary is that IPv6 on the host works fine, IPv6 in the container works fine if I change the adapter to macvlan, but not with veth. NDP seems to be OK so it may be some routing/sysctl issue?
Here’s more detail about the setup.
Host networking
Per Ryan Young | How to Assign IPv6 Addresses to LXD Containers on a VPS, so it looks like this:
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:16:3C:A8:DB:1B
inet addr:1.2.3.4 Bcast:0.0.0.0 Mask:255.255.255.192
inet6 addr: fe80::216:3cff:fea8:db1b/64 Scope:Link
inet6 addr: 2a00:1234:1:5678::abcd/128 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:437999 errors:0 dropped:0 overruns:0 frame:0
TX packets:24782 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:36212339 (34.5 MiB) TX bytes:7944323 (7.5 MiB)
$ ip -6 route show
2a00:1234:1::1 dev eth0 metric 1024 onlink pref medium
2a00:1234:1:5678::abcd dev eth0 proto kernel metric 256 pref medium
2a00:1234:1:5678::/64 dev lxcbr0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev lxcbr0 proto kernel metric 256 pref medium
fe80::/64 dev vethW2ZiF8 proto kernel metric 256 pref medium
default via 2a00:1234:1::1 dev eth0 metric 1024 pref medium
$ ping ipv6.google.com -c 3
PING ipv6.google.com (2a00:1450:400d:804::200e): 56 data bytes
64 bytes from 2a00:1450:400d:804::200e: seq=0 ttl=116 time=153.407 ms
64 bytes from 2a00:1450:400d:804::200e: seq=1 ttl=116 time=30.142 ms
64 bytes from 2a00:1450:400d:804::200e: seq=2 ttl=116 time=30.115 ms
--- ipv6.google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 30.115/71.221/153.407 ms
LXC with macvlan
Seems to be fine.
$ ip link add mvlan0 link eth0 type macvlan mode bridge
$ ifconfig mvlan0 up
$ cat /var/lib/lxc/v6test/config
...
lxc.net.2.type = macvlan
lxc.net.2.macvlan.mode = bridge
lxc.net.2.link = mvlan0
lxc.net.2.flags = up
lxc.net.2.hwaddr = 00:16:3e:2f:80:a8
$ lxc-start -n v6test
$ lxc-attach -n v6test
$ ifconfig eth2
eth2 Link encap:Ethernet HWaddr 00:16:3E:2F:80:A8
inet6 addr: 2a00:1234:1:5678:216:3eff:feef:88aa/64 Scope:Global
inet6 addr: fe80::216:3eff:feef:88aa/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:382 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:23541 (22.9 KiB) TX bytes:752 (752.0 B)
$ ip -6 route show
2a00:1234:1::1 dev eth2 metric 1024 pref medium
2a00:1234:1:5678::/64 dev eth2 proto kernel metric 256 pref medium
fe80::/64 dev eth2 proto kernel metric 256 pref medium
default via 2a00:1234:1::1 dev eth2 metric 1024 pref medium
$ ping ipv6.google.com -c 3
PING ipv6.google.com (2a00:1450:4017:80d::200e): 56 data bytes
64 bytes from 2a00:1450:4017:80d::200e: seq=0 ttl=118 time=0.656 ms
64 bytes from 2a00:1450:4017:80d::200e: seq=1 ttl=118 time=0.633 ms
64 bytes from 2a00:1450:4017:80d::200e: seq=2 ttl=118 time=0.746 ms
--- ipv6.google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.633/0.678/0.746 ms
LXC with veth
Here’s where the problem comes. Even though they are for LXD, I generally tried to follow thse resources: Can't get IPv6 Netmask 64 to work (no NAT, should be end to end), Getting universally routable IPv6 Addresses for your Linux Containers on Ubuntu 18.04 with LXD 4.0 on a VPS and Ryan Young | How to Assign IPv6 Addresses to LXD Containers on a VPS
I’ve assigned the /64 to lxcvbr0 (see above for the routes):
$ ifconfig lxcbr0
lxcbr0 Link encap:Ethernet HWaddr 00:16:3E:A7:17:58
inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0
inet6 addr: fe80::216:3eff:fea7:1758/64 Scope:Link
inet6 addr: 2a00:1234:1:5678::/64 Scope:Global
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:567 errors:0 dropped:0 overruns:0 frame:0
TX packets:604 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:42757 (41.7 KiB) TX bytes:997835 (974.4 KiB)
For the LXC container:
$ cat /var/lib/lxc/v6test/config
...
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:2f:80:a7
$ cat /etc/default/lxc
...
LXC_IPV6_ADDR="2a00:1234:1:5678::"
LXC_IPV6_MASK="64"
LXC_IPV6_NETWORK="${LXC_IPV6_ADDR}/${LXC_IPV6_MASK}"
$ lxc-attach -n v6test
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:16:3E:2F:80:A7
inet addr:10.0.3.122 Bcast:10.0.3.255 Mask:255.255.255.0
inet6 addr: 2a00:1234:1:5678:216:3eff:feef:80aa/64 Scope:Global
inet6 addr: fe80::216:3eff:feef:80aa/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1932 (1.8 KiB) TX bytes:1951 (1.9 KiB)
$ ip -6 route show
2a00:1234:1:5678::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::216:3eff:fea7:1758 dev eth0 proto ra metric 1024 expires 1739sec hoplimit 64 pref medium
Note that it seems to pull an address from dnsmasq perfectly OK. But:
$ ping ipv6.google.com -c 3
PING ipv6.google.com (2a00:1450:4001:831::200e): 56 data bytes
--- ipv6.google.com ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
Ping in the other direction (to the container) doesn’t seem to work either. I’ve set a NDP proxy on the host and it seems to be functional:
$ ip -6 neigh show proxy
2a00:1234:1:5678:216:3eff:feef:80aa dev eth0 proxy
$ tcpdump ip6 | grep 80aa
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:25:38.211704 IP6 fe80::3e61:40b:b8a4:1f7c > ff02::1:ff2f:80aa: ICMP6, neighbor solicitation, who has 2a00:1234:1:5678:216:3eff:feef:80aa, length 32
09:25:38.795718 IP6 fe80::216:3cff:fea8:db1b > fe80::3e61:40b:b8a4:1f7c: ICMP6, neighbor advertisement, tgt is 2a00:1234:1:5678:216:3eff:feef:80aa, length 32
09:25:39.326321 IP6 fe80::3e61:40b:b8a4:1f7c > ff02::1:ff2f:80aa: ICMP6, neighbor solicitation, who has 2a00:1234:1:5678:216:3eff:feef:80aa, length 32
09:25:39.915634 IP6 fe80::216:3cff:fea8:db1b > fe80::3e61:40b:b8a4:1f7c: ICMP6, neighbor advertisement, tgt is 2a00:1234:1:5678:216:3eff:feef:80aa, length 32
09:25:40.294708 IP6 fe80::3e61:40b:b8a4:1f7c > ff02::1:ff2f:80aa: ICMP6, neighbor solicitation, who has 2a00:1234:1:5678:216:3eff:feef:80aa, length 32
09:25:40.535611 IP6 fe80::216:3cff:fea8:db1b > fe80::3e61:40b:b8a4:1f7c: ICMP6, neighbor advertisement, tgt is 2a00:1234:1:5678:216:3eff:feef:80aa, length 32
^C86 packets captured
90 packets received by filter
0 packets dropped by kernel
Firewall shouldn’t be an issue, since I’ve just set ip6tables -P INPUT ACCEPT ; ip6tables -P FORWARD ACCEPT.
It could be a sysctl setting, but I think I’ve covered the obvious ones:
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.eth0.forwarding = 1
net.ipv6.conf.lxcbr0.forwarding = 1
net.ipv6.conf.all.proxy_ndp = 1
net.ipv6.conf.eth0.proxy_ndp = 1
Do you have any ideas? NDP? Routing? Sysctl? Thoughts appreciated!