I reaching out to you because I need your advice related to my network configuration on my server. I have installed LXD 4.0 and I noticed that this version comes with network routed. In the past I had to make a Bridge (Host-Shared Bridge) on my network card to provide Public IPs to my containers since MACVLAN can’t communicate containers with Host and vice versa.
Now I want to make more efficient my configuration taking advantage of this new concept (routed)
Before to do that I have some questions:
routed networking is more efficient than host-shared Bridge referring to resources consumption and speed?
Do I have to make extra rules on IPtables in my Host to forward the inbound traffic from Internet to reach each container?
When I use routed mode what is the advantage to share the host’s MAC address? How about Bridge mode, this doesn’t?
Can Bridge and routed coexist on the same server to provide Bridge mode to Windows Machines and routed mode to containers?
Can I use this new concept to provide routed mode to KVM Machines?
Routed mode use a veth pair to connect the container to the host. This is similar to bridged or p2p NIC types, but without the need for a virtual switch in bridged mode. I would expect the performance to be similar or better than bridged mode, especially for traffic going between external network and the container as there are less ‘moving parts’ (i.e no virtual ethernet bridge). That being said it is likely to be less performant than ipvlan type as each packet will be processed by the host’s routing table. But it allows container to host communication, unlike ipvlan.
This depends on whether you have any existing rules blocking traffic to your external IPs that you are passing into your container. In general the host will process these rules using the FORWARD table of your firewall (as it is routing packets), so checking the existing ruleset would be a good idea.
Routed mode is useful when you having 1 or more external IPs that you want to use directly inside a container but your ISP is not allowing you to use multiple MAC addresses on the external interface. In comparison, when using bridged mode connected to the host’s external port, each container will appear on the network as a separate device (with its own MAC address). Some ISPs do not allow this. The other difference is that in routed mode, the LXD administrator is in charge of assigning IPs to containers, whereas in bridged mode in its default configuration, containers can pick their own IPs (this can be restricted using the security.*_filtering settings on bridged NIC (https://linuxcontainers.org/lxd/docs/master/instances#nictype-bridged).
Yes absolutely. You can mix and match as you need to. You can even connect multiple NIC devices to a container, e.g. one routed and one bridged.
routed mode doesn’t currently work with LXD VMs as it relies on being able to configure IPs on the interface before the container starts. Although there is nothing inherently preventing this working for VMs (with some manual setup), so maybe something we add in the future.
Thank you Thomas for taking the time to resolve all my questions. Now I have better understanding on how that works.
Just a final doubt.
There are typically two ways to create a bridged network setup:
host-shared bridge: create a bridge out of your main network interface which will hold both the host’s IP and the container’s IP addresses.
independent bridge: create a different bridge out of thin air and link your containers together on this bridge, but use forwarding to get it out on the internet or to get traffic into it.
When I use LXD containers, which bridge is better?
Neither is better, they are used for different things.
Joining your containers to the external network with a bridge that includes the external interface can be useful if you want your containers to appear as their own nodes on the network, with their own MAC addresses. And then they can share the DHCP and IPv6 router advertisement services on the wider network too. This can also be useful if you want your containers to provide services that rely on broadcast traffic to the wider network.
Using a private bridge, and using NAT to allow outbound traffic from the bridge is a quick and reliable way to get going quickly, which is why it is the LXD default networking option.
You can then optionally using a proxy device to selectively forward ports on the host’s IP(s) into specific containers to publish services if needed.
However it can become cumbersome if you want to pass external IPs into the containers (rather than using NAT & port forwarding).