Thats you’re problem there, you’ve got the same route to two different interfaces, which means the DNS responses are going awry.
Your test resolution from the fan networking to 10.1.0.1 worked, which mean that dnsmasq on lxdbr0 must be working. That’s what gave the game away.
You need to reduce the size of your two networks, lxdbr0 and dmz0 and make sure they dont overlap.
Thanks.
I find networking very frustrating. I can haven’t a good mental model of how all the bits fit together to make a working system, so find both the configuration and the problem tracing difficult.
My problem, but a good guide written by a non-networking person would help me.
Do you know of any?
I personally found this to be a good way to present the basics of networking recently, although it goes to the very basic physical level initially.
In this case your dmz0 interface is a bit of a problem, because it has addresses on it from 4 different Layer 3 (L3) subnets; 192.168.157.215/24, 10.1.0.215/16, 10.2.0.215/16, and 10.3.0.215/16.
This scenario is mentioned in the link above in the section " L3 to L2 segments relationship".
The number after the / indicates the size of the subnet (i.e the addresses that can be reached on the same L2 segment without needing to go through a router).
So when adding an address of 10.1.0.215/16, to dmz0, you’re telling the OS that the rest of that subnet (10.1.0.0/16) is directly reachable via that interface. So if a packet is generated with an address in that network it will go out of that interface.
Adding another interface with an address in the same subnet (in this case lxdbr0 with 10.1.0.1/16) will cause another route to be added saying that 10.1.0.0/16 is directly reachable via lxdbr0 interface.
You now have 2 routes to the same destination subnet down two different interfaces. But a packet can only go down one interface, so the OS picks one, and either way you slice it, sometimes this will be the wrong one.
What purpose does the dmz0 internface serve?
It’s the physical interface through which the servers are connected. I suspect each subnet needs to be on a separate vlan as they don’t need to talk to each other.
There’s an issue in the core of this: LXD’s cluster networking has a subnet which extends over multiple servers, so random IP Addresses are found in different, physically separated places. Perhaps MACVLAN can do this for me, although I’ll need to use a local DNS service on a unix socket mapped into the containers and static IP Addresses to make this work - neither DHCP nor DNS resolution are present on the DMZ network.
A hassle but simpler (I think) than configuring flanel - unless you think flanel will be simpler & more manageable?
Or, I could switch them all to fan networks. I found that easy enough to get working 
Are you wanting the ability for instances on one subnet can communicate with instances on another subnet?
I’m confused why there are multiple subnets involved. Why not just have one large fan network?
Also, fan networking is an overlay network, meaning it is layered ontop of the physical network, so its not really expected that other subnets can communicate with it (which is why by default LXD will nat outbound traffic to the host’s external IP address).
I’ve 3 different sets of containers for different purposes, so I put them in 3 subnets - as I thouught I can use the firewall to keep them separate simply.
If I use 1 big fan network then I have to start tagging containers and then … how to manage which connections are permitted and which blocked? N * (N-1) communication paths to manage?
I’ve ingress & egress gateways for the fan network which is working … simple enough to replicate. I can’t think of any interconnects needed between the subnets, but the future is better not constrained unnecessarily. Interconnect through the ingress / egress gateways would be OK and is manageable.
So if you have 3 different fan networks, each will have its own bridge on each LXD host, and the routing table will allow routing between different subnets.
So I still don’t see why you need the fan subnets exposed onto the dmz0 interface?
I’m not familiar with the terms ingress and egress gateways (in a LXD context at least), so I don’t know what those are.
These servers (I was only using one for the test for simplicity) are in a cluster … the dmz0 I/F is connected to the DMZ switch through which they’re inter-connected, and from which they can reach the big world.
The ingress and egress gateways are also described as reverse proxys & (normal) proxys, i.e. they allow boxes outside the LXD realm to connect to services provided by containers within (with authorisation and authentication), and they allow (limited, managed) access from within to services outside, principally images & (update) software download. This is a better solution (easier to manage) to egress than NAT - in fact, once these gateways are in place and configured, NAT is no longer required if I use fan networks. Big world DNS resolution could also be provided from the egress proxy, although I haven’t implemented that yet.
The ingress gateways are for other systems which need to use services provided from containers in LXD. There is existing (limited) routing from the internal networks to the DMZ network.
You cannot expose the fan networks into the external network via dmz at Layer 2, as it will cause routing conflicts as you have seen.
You either need to remove the fan and just join the instances onto the external network (via a normal bridge connected to dmz0 or a macvlan using dmz0 as the parent), or keep the fan and just use it for internal traffic, and add an additional NIC device to the instances (using macvlan or bridge connect to dmz0) for the external network access.
You may also want to look at our recently added OVN support, as this allows creating multi-node virtual overlay networks, with distributed routers that expose a single external routing address onto the external network (that fails over between nodes), allowing for your upstream gateways to specify a route target for the virtual network’s subnet.
Yes, I have (?had) the fan network working with ingress and egress gateways. Seems to be good for my purposes. An LXD proxy device gets the traffic from dmz0 into the ingress gateway, from where it can be redirected (by NGINX) to where it’s needed.
I was trying to avoid OVN or similar networking as it’s another learning curve to climb, if you’ll pardon a mangled metaphor.
So where do these subnets come into the mix, as these are the ones causing the issue with DNS, because of the overlapping routes of 10.1.0.0/16 (I presume the other 2 are similarly defined on other nodes)?
10.1.0.0/16 dev dmz0 proto kernel scope link src 10.1.0.215
10.2.0.0/16 dev dmz0 proto kernel scope link src 10.2.0.215
10.3.0.0/16 dev dmz0 proto kernel scope link src 10.3.0.215
If each subnet also belongs to an equivalent local lxdbr0 interface on each LXD node, why don’t you add static routes to each node and your gateways for each of these subnets pointing towards the external IP of each LXD host, that way you won’t need them to be assigned to the dmz0 interface, but the other LXD nodes and your gateways will still know how to reach those addresses, at layer 3 rather than layer 2.
10.2.0.0/16 is the underlay subnet for the (working) fan network.
10.1.0.0/16 was an LXD bridge network - I’m going to convert it to be the underlay network for another fan network. Then, there’s a single IP address for it on the dmz0 for each host and routing is simple.
10.3.0.0/16 was a macvlan network, which will be the service subnet for a kubernetes cluster. That should be OK as there are few services exposed (so they won’t overload the NICs) and they can have fixed IP addresses on the NIC.
I’m assuming the LXD macvlan mode is (or can be configured to be) ‘bridge’. If not, I may just need to use (another) fan network.
Or, as you’re suggesting, I can learn about OVN networks.
Hmmm … thinking about it, kubernetes won’t work with the DHCP on a fan network … it does its own. So, macvlan for that, or some variety of OVN.
Thanks again @tomp. DNS resolution now working on all 3 subnets. macvlan subnet working, although setting IP Address is a bit of a pain - lxc config device add XXXX eth0 nic network=k8sbr0 then lxc config device set XXXX eth0 ipv4.address=10.3.3.3 isn’t working so I’m doing it directly on the eth0 device within the container. 
I’ve got the syntax wrong (see below).
A bit messy, but functional. 
Enough for 1 day.
… just found instructions. it should be ipv4.address 10.3.3.3 without the = 