I was thinking about this and I believe the easiest way to create private subnets which allow ethernet communication between different containers without involving the host is to use a dummy
device on the host for each subnet and then added macvlan
NICs to each container using the relevant dummy
device as the parent, e.g.
Lets create a router
container that has its eth0
connected to the default lxdbr0
managed bridge (for external connectivity to the host and beyond).
Then lets setup subnet1
and connect the router
and a subnet1-c1
container together, and configure the subnet1-c1
container to use router
as its default gateway.
First, create the router (assuming that the default protocol is configured to connect eth0 to the lxdbr0 bridge):
lxc init images:ubuntu/focal router
lxc config show router --expanded
...
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
...
Now lets create the dummy device for subnet1 on the host and create a LXD macvlan network to reference it:
sudo ip link add subnet1 type dummy
sudo ip link set subnet1 up
lxc network create subnet1 --type=macvlan parent=subnet1
Now lets connect router
to subnet1
using a new eth1
NIC:
lxc config device add router eth1 nic network=subnet1
Start the router
container and configure a manual IP inside the container on eth1
.
lxc start router
lxc exec router -- ip a add 10.0.0.1/24 dev eth1
lxc ls router
+--------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+--------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| router | RUNNING | 10.98.30.9 (eth0) | fd42:f402:8623:5b6b:216:3eff:fe32:b4ab (eth0) | CONTAINER | 0 |
| | | 10.0.0.1 (eth1) | | | |
+--------+---------+-------------------+-----------------------------------------------+-----------+-----------+
Now lets create a new container called subnet1-c1
that only has its eth0
connected to subnet1
(i.e it has no external connectivity).
lxc init images:ubuntu/focal subnet1-c1
lxc config device add subnet1-c1 eth0 nic network=subnet1
lxc config show subnet1-c1 --expanded
...
devices:
eth0:
network: subnet1
type: nic
...
Start the container and then setup its IP manually, and check connectivity to router
.
lxc start subnet1-c1
lxc exec subnet1-c1 -- ip a add 10.0.0.2/24 dev eth0
lxc exec subnet1-c1 -- ping -c1 10.0.0.1
If that is working then you have basic subnet connectivity up and running.
Now to get the router
container operation as a router, and informing subnet1-c1
to use it as its default gateway.
Lets enable IPv4 forwarding and add a SNAT rule to the router
container:
lxc exec router -- sysctl sysctl net.ipv4.ip_forward=1
lxc exec router -- apt install iptables -y
lxc exec router -- iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
Now instruct subnet1-c1
to use router
for its default gateway and check external connectivity.
lxc exec subnet1-c1 -- ip r add default 10.0.0.1 dev eth0
lxc exec subnet1-c1 -- ping -c1 8.8.8.8