Fan Networking creation?

tatric · February 18, 2019, 2:02pm

I have a 3 host LXD cluster, with no container networking and a default profile, Snap LXD 3.10
I want to create a network where all containers can talk/ping each other by name or ip address.

I issued the command to prepare the 3 hosts:
stev@de-db01:~$ sudo lxc network create fan0 --target=de-db01
Network fan0 pending on member de-db01
stev@de-db01:~$ sudo lxc network create fan0 --target=de-db02
Network fan0 pending on member de-db02
stev@de-db01:~$ sudo lxc network create fan0 --target=de-db03
Network fan0 pending on member de-db03
stev@de-db01:~$ sudo lxc network create fan0 bridge.mode=fan
Network fan0 created

stev@de-db01:~$ sudo lxc network attach-profile fan0 default

stev@de-db01:~$ sudo lxc network list
±-------±---------±--------±------------±--------±--------+
| NAME | TYPE | MANAGED | DESCRIPTION | USED BY | STATE |
±-------±---------±--------±------------±--------±--------+
| ens160 | physical | NO | | 0 | |
±-------±---------±--------±------------±--------±--------+
| fan0 | bridge | YES | | 0 | CREATED |
±-------±---------±--------±------------±--------±--------+

Now before DNS and the containers ip address get fixed (reboots) the hosts should be able to communicate on the fan network 240.X.0.1 where X is the address of the server on the “real” network?

ip a s shows 3 new fan network adaptors (fan0 fan0-mtu and fan0-fan) and the routes have been added to the routing table:
stev@de-db01:~$ ip r s
default via 192.168.81.254 dev ens160 proto static
192.168.81.0/24 dev ens160 proto kernel scope link src 192.168.81.4
240.0.0.0/8 dev fan0 proto kernel scope link src 240.4.0.1

pings to de-db02 work via this underlay network… good
stev@de-db01:~$ ping 240.9.0.1
PING 240.9.0.1 (240.9.0.1) 56(84) bytes of data.
64 bytes from 240.9.0.1: icmp_seq=1 ttl=64 time=0.551 ms
64 bytes from 240.9.0.1: icmp_seq=2 ttl=64 time=0.321 ms
64 bytes from 240.9.0.1: icmp_seq=3 ttl=64 time=0.333 ms

But pings to de-db03 fail
stev@de-db01:~$ ping 240.22.0.1
PING 240.22.0.1 (240.22.0.1) 56(84) bytes of data.
From 240.4.0.1 icmp_seq=1 Destination Host Unreachable
From 240.4.0.1 icmp_seq=2 Destination Host Unreachable
From 240.4.0.1 icmp_seq=3 Destination Host Unreachable
From 240.4.0.1 icmp_seq=4 Destination Host Unreachable

stev@de-db01:~$ arp
Address HWtype HWaddress Flags Mask Iface
240.9.0.1 ether 7e:10:67:39:68:7a C fan0
240.22.0.1 (incomplete) fan0
…

when looking at the other cluster members
de-db02 works fine to both de-db01 and de-db03, arp table full.
but de-db03 is the same as de-db01 in that its all setup with network adaptors, but can’t ping and doesn’t have a mac address for de-db01 in its arp table.

If I reboot everything then then the containers get ip address and the containers on de-db02 can ping the containers on de-db01 and de-db03 and its-self, but containers on de-db01 can’t ping containers on de-db03 and vice-versa, of course.

So is this me doing it wrong or is it a bug? Is there anything further i should do/try?

stgraber · February 18, 2019, 11:31pm

Can you confirm that all your hosts or on the same subnet and that this is configured the same way on all of them? That is, they’re using the same netmask on all of them and ideally same default gateway?

I’m assuming you confirmed that they also can all ping each-other over their non-fan IP addresses?

tatric · February 19, 2019, 9:36am

Yes sorry forgot the obvious. All hosts have same network settings, and can ping each other fine via the normal network and the cluster is working fine.

de-db01 192.168.81.4/24
de-bd02 192.168.81.9/24
de-db03 192.168.81.22/24
and all have same gateway.192.168.81.255
All firewalls are off

This is the arp table on the other server de-db03 as you can see de-db01 fan0 mac address is missing, therefore i can’t ping it etc. (i have removed all the other entries in )
stev@de-db03:~$ arp
Address HWtype HWaddress Flags Mask Iface
240.9.0.1 ether 7e:10:67:39:68:7a C fan0
240.4.0.1 (incomplete) fan0
…

But the strange thing is this works fine on host de-db02.
stev@de-db02:~$ arp
Address HWtype HWaddress Flags Mask Iface
…
240.22.0.1 ether a2:58:b8:14:5e:fb C fan0
240.4.0.1 ether 5e:1b:d5:98:13:72 C fan0

I have removed and rebuilt the fan network 3 times and it always creates this way( a missing mac in arp table on de-db01 and de-db03 for fan network), with no errors (i ran the lxc network create fan0 … on de-db01)

Any help is appreciated.

stgraber · February 19, 2019, 3:59pm

Are you maybe missing the route for the fan bridge somehow?

ip -4 route show

tatric · February 19, 2019, 4:31pm

No,it has been added on all servers.

de-db01 is shown in the 1st post, but i’ll show again for completeness.

stev@de-db01:~$ ip -4 route show
default via 192.168.81.254 dev ens160 proto static
192.168.81.0/24 dev ens160 proto kernel scope link src 192.168.81.4
240.0.0.0/8 dev fan0 proto kernel scope link src 240.4.0.1

stev@de-db03:~$ ip -4 route show
default via 192.168.81.254 dev ens160 proto static
192.168.81.0/24 dev ens160 proto kernel scope link src 192.168.81.22
240.0.0.0/8 dev fan0 proto kernel scope link src 240.22.0.1

stev@de-db02:~$ ip -4 route show
default via 192.168.81.254 dev ens160 proto static
192.168.81.0/24 dev ens160 proto kernel scope link src 192.168.81.9
240.0.0.0/8 dev fan0 proto kernel scope link src 240.9.0.1

Routes have been added.

stgraber · February 19, 2019, 5:02pm

Yeah, that looks fine. Can you check that all fan interfaces are properly up on all hosts and that there’s no firewalling in place which would block the vxlan traffic?

After that, my next guess would involve tcpdump to see what’s actually hitting the network.

tatric · February 21, 2019, 3:21pm

Below is the tcpdumps when I log into another terminal and issue pings to the other cluster members fan addresses on same host.

when i ‘ping 240.22.0.1’ (de-db03 which fails)
> @de-db01:~$ sudo tcpdump -i fan0

    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on fan0, link-type EN10MB (Ethernet), capture size 262144 bytes
    11:13:26.768619 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    11:13:27.790136 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    11:13:28.810106 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    11:13:30.770326 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    11:13:31.786105 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    11:13:32.810120 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    11:13:33.834188 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    11:13:34.858108 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    11:13:35.882101 ARP, Request who-has 240.22.0.1 tell de-db01, length 28
    ^C
    9 packets captured
    9 packets received by filter
    0 packets dropped by kernel

When I ‘ping 240.9.0.1’ (de-db02 which works)
> @de-db01:~$ sudo tcpdump -i fan0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on fan0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:15:16.865860 IP de-db01 > 240.9.0.1: ICMP echo request, id 29161, seq 1, length 64
11:15:16.866220 IP 240.9.0.1 > de-db01: ICMP echo reply, id 29161, seq 1, length 64
11:15:17.865338 IP de-db01 > 240.9.0.1: ICMP echo request, id 29161, seq 2, length 64
11:15:17.865596 IP 240.9.0.1 > de-db01: ICMP echo reply, id 29161, seq 2, length 64
11:15:18.864865 IP de-db01 > 240.9.0.1: ICMP echo request, id 29161, seq 3, length 64
11:15:18.865147 IP 240.9.0.1 > de-db01: ICMP echo reply, id 29161, seq 3, length 64
11:15:19.864567 IP de-db01 > 240.9.0.1: ICMP echo request, id 29161, seq 4, length 64
11:15:19.864892 IP 240.9.0.1 > de-db01: ICMP echo reply, id 29161, seq 4, length 64
11:15:20.864628 IP de-db01 > 240.9.0.1: ICMP echo request, id 29161, seq 5, length 64
11:15:20.864945 IP 240.9.0.1 > de-db01: ICMP echo reply, id 29161, seq 5, length 64
11:15:21.874624 ARP, Request who-has de-db01 tell 240.9.0.1, length 28
11:15:21.874642 ARP, Reply de-db01 is-at 0e:68:12:13:61:34 (oui Unknown), length 28
11:15:22.026108 ARP, Request who-has 240.9.0.1 tell de-db01, length 28
11:15:22.026403 ARP, Reply 240.9.0.1 is-at 66:df:6c:f6:a8:45 (oui Unknown), length 28
^C
14 packets captured
14 packets received by filter
0 packets dropped by kernel

Same if i capture info on the fan0-fan network interface.

If i try again across the network de-db03 to de-db01 then nothing is received at tcpdump
if i try from de-db02 then a it works as expected and tcpdump captures the same info.

If I ‘snap stop lxd’ then ‘ip a s’ all the fan interfaces have gone and ‘snap start lxd’ recreates them all again, but they don’t work. Nothing bad about networks just “Initializing networks” in /var/snap/lxd/common/lxd/logs/lxd.log

Are the iptable rules correct?
> sudo iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain /* generated for LXD network fan0 */
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain /* generated for LXD network fan0 */
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps /* generated for LXD network fan0 */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             /* generated for LXD network fan0 */
ACCEPT     all  --  anywhere             anywhere             /* generated for LXD network fan0 */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:domain /* generated for LXD network fan0 */
ACCEPT     udp  --  anywhere             anywhere             udp spt:domain /* generated for LXD network fan0 */
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps /* generated for LXD network fan0 */

stgraber · February 22, 2019, 3:56pm

iptables looks fine, nothing’s blocked.

tcpdump would be more useful if you were dumping the underlay device (physical device going out of the machine) as the problem appears to be at that level, so it’d be interesting to see if there’s anything obviously wrong going on with VXLAN there.