How to get two containers with ipv6 to talk to eachother on separate bridges

Incus
1

I want to test ipv6 with two containers in separate networks within the same project, within the same lxd-host.

Containers are “one” and “two”.

My two networks are configured as:

BR001 - configured as:

config:
  ipv4.address: 10.241.98.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:d264:4416:5fe8::1/64
  ipv6.nat: "true"
description: ""
name: BR001
type: bridge
used_by:
- /1.0/instances/one
managed: true
status: Created
locations:
- none

The other network BR002

config:
  ipv4.address: 10.221.208.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:d264:4416:5fe8::2/64
  ipv6.nat: "true"
description: ""
name: BR002
type: bridge
used_by:
- /1.0/instances/two
managed: true
status: Created
locations:
- none

The instances can ping their gateways:

root@one:~# ping6 fd42:d264:4416:5fe8::1
PING fd42:d264:4416:5fe8::1(fd42:d264:4416:5fe8::1) 56 data bytes
64 bytes from fd42:d264:4416:5fe8::1: icmp_seq=1 ttl=64 time=0.121 ms

root@two:~# ping6 fd42:d264:4416:5fe8::2
PING fd42:d264:4416:5fe8::2(fd42:d264:4416:5fe8::2) 56 data bytes
64 bytes from fd42:d264:4416:5fe8::2: icmp_seq=1 ttl=64 time=0.149 ms

However, I can’t seem to get the containers to communicate in-between.

root@one:~# ping6 fd42:d264:4416:5fe8:216:3eff:feab:f109
PING fd42:d264:4416:5fe8:216:3eff:feab:f109(fd42:d264:4416:5fe8:216:3eff:feab:f109) 56 data bytes
^C
— fd42:d264:4416:5fe8:216:3eff:feab:f109 ping statistics —
2 packets transmitted, 0 received, 100% packet loss, time 1031ms

I am not a ipv6 guru, so I suspect I’m doing something wrong with either:

  • How my networks need to be configured
  • How my host is setup
  • Something else?

Anyone that can advice here?

2

Ah, so the problem here is that your two networks are overlapping.
You should probably use something like:

  • ipv6.address: fd42:d264:4416:5fe8::1/64
  • ipv6.address: fd42:d264:4416:5fe9::1/64

So that you’re dealing with two distinct networks instead.

3

I reconfigured the BR002 and can now ping6 the gateways:

root@one:~# ping6 fd42:d264:4416:5fe8::1
PING fd42:d264:4416:5fe8::1(fd42:d264:4416:5fe8::1) 56 data bytes
64 bytes from fd42:d264:4416:5fe8::1: icmp_seq=1 ttl=64 time=0.130 ms

root@one:~# ping6 fd42:d264:4416:5fe9::1
PING fd42:d264:4416:5fe9::1(fd42:d264:4416:5fe9::1) 56 data bytes
64 bytes from fd42:d264:4416:5fe9::1: icmp_seq=1 ttl=64 time=0.136 ms

But I can’t ping the container(s)

root@one:~# ping6 fd42:d264:4416:5fe9:216:3eff:fe16:4118
PING fd42:d264:4416:5fe9:216:3eff:fe16:4118(fd42:d264:4416:5fe9:216:3eff:fe16:4118) 56 data bytes
^C
— fd42:d264:4416:5fe9:216:3eff:fe16:4118 ping statistics —
2 packets transmitted, 0 received, 100% packet loss, time 1012ms

4

Can you ping both containers from the host?

Might be worth checking for any firewall at that point.

Stéphane

5

@stgraber I can also ping their respective ipv4 addresses:

+----------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| one            | RUNNING | 10.241.98.26 (eth0)   | fd42:d264:4416:5fe8:216:3eff:fe9c:b394 (eth0) | CONTAINER | 0         |
+----------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| two            | RUNNING | 10.221.208.100 (eth0) | fd42:d264:4416:5fe9:216:3eff:fe16:4118 (eth0) | CONTAINER | 0         |
+----------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+

root@one:~# ping 10.221.208.100
PING 10.221.208.100 (10.221.208.100) 56(84) bytes of data.
64 bytes from 10.221.208.100: icmp_seq=1 ttl=63 time=0.166 ms

root@two:~# ping 10.241.98.26
PING 10.241.98.26 (10.241.98.26) 56(84) bytes of data.
64 bytes from 10.241.98.26: icmp_seq=1 ttl=63 time=0.181 ms

Could this be something with ip6 forwarding or iptables?

I did just now:

sudo sysctl -w net.ipv6.conf.all.forwarding=1

and iptable6 shows that the default policy is ACCEPT, so it shouldn’t be a firewall issue…

sudo ip6tables -L -n

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Still no luck with the ipv6 ping6 between containers.

6

Can you show ‘nft list ruleset’?

Modern systems don’t use iptables anymore.

Stéphane

7

erik@iceberg:~$
erik@iceberg:~$ nft list ruleset
erik@iceberg:~$

8

@stgraber My bad ^

Here is the output (sudo)

root@iceberg:~# nft list ruleset

table inet lxd {
	chain pstrt.BR001 {
		type nat hook postrouting priority srcnat; policy accept;
		@nh,96,24 717154 @nh,128,24 != 717154 masquerade
		@nh,64,64 18249379968139550696 @nh,192,64 != 18249379968139550696 masquerade
	}

	chain fwd.BR001 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "BR001" accept
		ip version 4 iifname "BR001" accept
		ip6 version 6 oifname "BR001" accept
		ip6 version 6 iifname "BR001" accept
	}

	chain in.BR001 {
		type filter hook input priority filter; policy accept;
		iifname "BR001" tcp dport 53 accept
		iifname "BR001" udp dport 53 accept
		iifname "BR001" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		iifname "BR001" udp dport 67 accept
		iifname "BR001" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
		iifname "BR001" udp dport 547 accept
	}

	chain out.BR001 {
		type filter hook output priority filter; policy accept;
		oifname "BR001" tcp sport 53 accept
		oifname "BR001" udp sport 53 accept
		oifname "BR001" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		oifname "BR001" udp sport 67 accept
		oifname "BR001" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
		oifname "BR001" udp sport 547 accept
	}

	chain pstrt.BR002 {
		type nat hook postrouting priority srcnat; policy accept;
		@nh,96,24 712144 @nh,128,24 != 712144 masquerade
		@nh,64,64 18249379968139550697 @nh,192,64 != 18249379968139550697 masquerade
	}

	chain fwd.BR002 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "BR002" accept
		ip version 4 iifname "BR002" accept
		ip6 version 6 oifname "BR002" accept
		ip6 version 6 iifname "BR002" accept
	}

	chain in.BR002 {
		type filter hook input priority filter; policy accept;
		iifname "BR002" tcp dport 53 accept
		iifname "BR002" udp dport 53 accept
		iifname "BR002" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		iifname "BR002" udp dport 67 accept
		iifname "BR002" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
		iifname "BR002" udp dport 547 accept
	}

	chain out.BR002 {
		type filter hook output priority filter; policy accept;
		oifname "BR002" tcp sport 53 accept
		oifname "BR002" udp sport 53 accept
		oifname "BR002" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		oifname "BR002" udp sport 67 accept
		oifname "BR002" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
		oifname "BR002" udp sport 547 accept
	}
}

root@iceberg:~#

9

Not seeing anything that would block that traffic.

10 
stgraber@dakara:~$ incus network create testbr0 ipv6.address=fd42:d264:4416:5fe8::1/64 ipv6.nat=true ipv4.address=auto ipv4.nat=true
Network testbr0 created
stgraber@dakara:~$ incus network create testbr1 ipv6.address=fd42:d264:4416:5fe9::1/64 ipv6.nat=true ipv4.address=auto ipv4.nat=true
Network testbr1 created
stgraber@dakara:~$ incus launch images:ubuntu/22.04 u1 --network testbr0
Creating u1
Starting u1
stgraber@dakara:~$ incus launch images:ubuntu/22.04 u2 --network testbr1
Creating u2
Starting u2
stgraber@dakara:~$ incus list u
+------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4          |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| u1   | RUNNING | 10.153.249.50 (eth0)  | fd42:d264:4416:5fe8:216:3eff:fef5:dc9 (eth0)  | CONTAINER | 0         |
+------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| u2   | RUNNING | 10.145.217.188 (eth0) | fd42:d264:4416:5fe9:216:3eff:fe6e:f542 (eth0) | CONTAINER | 0         |
+------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
stgraber@dakara:~$ incus exec u1 bash
root@u1:~# ping6 -n fd42:d264:4416:5fe9:216:3eff:fe6e:f542
PING fd42:d264:4416:5fe9:216:3eff:fe6e:f542(fd42:d264:4416:5fe9:216:3eff:fe6e:f542) 56 data bytes
64 bytes from fd42:d264:4416:5fe9:216:3eff:fe6e:f542: icmp_seq=1 ttl=63 time=0.064 ms
64 bytes from fd42:d264:4416:5fe9:216:3eff:fe6e:f542: icmp_seq=2 ttl=63 time=0.066 ms
^C
--- fd42:d264:4416:5fe9:216:3eff:fe6e:f542 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1025ms
rtt min/avg/max/mdev = 0.064/0.065/0.066/0.001 ms
root@u1:~# 
exit
stgraber@dakara:~$ incus exec u2 bash
root@u2:~# ping6 -n fd42:d264:4416:5fe8:216:3eff:fef5:dc9
PING fd42:d264:4416:5fe8:216:3eff:fef5:dc9(fd42:d264:4416:5fe8:216:3eff:fef5:dc9) 56 data bytes
64 bytes from fd42:d264:4416:5fe8:216:3eff:fef5:dc9: icmp_seq=1 ttl=63 time=0.056 ms
^C
--- fd42:d264:4416:5fe8:216:3eff:fef5:dc9 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.056/0.056/0.056/0.000 ms
root@u2:~# 
exit
stgraber@dakara:~$
1 Like
11

@stgraber I may have made some mistake about the actual ipv6 addresses when pinging the containers. Its working now.

ping6 of container two:

root@one:~# ping6 fd42:d264:4416:5fe9:216:3eff:fe16:4118
PING fd42:d264:4416:5fe9:216:3eff:fe16:4118(fd42:d264:4416:5fe9:216:3eff:fe16:4118) 56 data bytes
64 bytes from fd42:d264:4416:5fe9:216:3eff:fe16:4118: icmp_seq=1 ttl=63 time=0.181 ms
^C
— fd42:d264:4416:5fe9:216:3eff:fe16:4118 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.145/0.163/0.181/0.018 ms

ping6 of container one:

root@two:~# ping6 fd42:d264:4416:5fe8:216:3eff:fe9c:b394
PING fd42:d264:4416:5fe8:216:3eff:fe9c:b394(fd42:d264:4416:5fe8:216:3eff:fe9c:b394) 56 data bytes
64 bytes from fd42:d264:4416:5fe8:216:3eff:fe9c:b394: icmp_seq=1 ttl=63 time=0.166 ms

Thanx alot!

My next step from this is to figure out how to setup a testing router (toredo?) to act at as a fake ipv6 gateway I guess and see if I can somehow get this to work as I want my containers on ipv6 down the road.