I have a lxd cluster with 2 nodes running on centos 7.7, the other nodes running on ubuntu 18.04.
From the ubuntu nodes or the containers running on ubuntu nodes, I can ping successfully the containers running on the centos nodes. The centos nodes containers can access no where except the containers in same node.
On ubuntu nodes,
ip -d link show lxdfan0-fan outputs:
5: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue
master lxdfan0 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 32:4f:d9:8b:62:b0 brd ff:ff:ff:ff:ff:ff promiscuity 1
vxlan id 15728640
fan-map 240.0.0.0/8:192.144.0.0/16 local 192.144.60.17 dev eth0 ...
Please notice the fan-map 240.0.0.0/8:192.144.0.0/16.
On centos nodes,
ip -d link show lxdfan0-fan outputs:
ip -d link show lxdfan0-fan
7: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue
master lxdfan0 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 2e:89:be:7e:26:bd brd ff:ff:ff:ff:ff:ff promiscuity 1
vxlan id 15728640 local 192.144.60.16 dev eth0 ...
Please notice fan-map not there.
When I ran ip link change dev lxdfan0-fan type vxlan fan-map 240.0.0.0/8:192.144.0.0/16 on centos nodes, below error being found:
vxlan: unknown command “fan-map”?
lxd is running version 3.18 snap.
I tried two centos kernel versions:
4.4.207-1.el7.elrepo.x86_64 #1 SMP Sat Dec 21 08:00:19 EST 2019 x86_64 x86_64 x86_64 GNU/Linux
3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
I’m surprised that you got that far, I would have expected things to fail in a much clearer way.
The issue is that the kernel fan feature itself isn’t mainline. It’s currently an Ubuntu specific patch so running a non-Ubuntu kernel or one which wasn’t patched in the same way will lead to problems.
We have logic in LXD to only offer the fan on suitable kernels but the mixed cluster case is something that’s defeating that particular check (so long as you created the fan bridge on an Ubuntu node).
Is it possible to have work around provided? I can apply kernel patch or route2 patch. I can do anything in case it works
The centos nodes in the mixed cluster are very useful for our workload. Some testing workloads require centos with lower kernel version to run. So in our gitab runner lxd executor, we specify the target to run them. I think it’s a common use case.
Those look like the main two patches to apply to get the kernel side functionality.
If you’re using the LXD snap then that’s all you should need as iproute comes from the snap and so already has the needed support. If you’re not using the snap, the iproute2 patch may be found in https://patches.ubuntu.com/i/iproute2/
Note that there is no reason for any of this to be Ubuntu-specific and I’d be more than happy to see someone take a stab at pushing this subnet mapping feature upstream in some generic form. It’s on our own backlog for @brauner to take a look at some point, but we’re not planning to take that on until the second part of this year at least.
