I’m fairly new to lxd and so it’s possible that I am missing something obvious here. If so, my apologies and I would very much appreciate any points.
The Setup
I have a number of Raspberry Pi 3s and 4s running Ubuntu 20.10 (mix of 64 and 32 bit) attached to a VLAN-capable hardware switch.
Each Pi is member of a single lxd cluster, using a dedicated VLAN interface for clustering (i.e. each Pi has an eth0 in the default VLAN with a DHCP-provided IP and a vlan2 interface in VLAN 2 with a hard-coded IP from 10.0.0.0/24).
The Plan
I would like to be able to launch lxc containers on any host through lxd clustering with sets of these containers being placed in dedicated VLANs so as to create a number of isolated networking environments for them.
For example, I might have containers foo, bar and baz running on pi1 and pi2 sharing VLAN ID 5 and at the same time also have containers bam and bat, running on pi2 and pi3 sharing a separate VLAN 6:
| container | lxd host | vlan id |
|---|---|---|
| foo | pi1 | 5 |
| bar | pi1 | 5 |
| baz | pi2 | 5 |
| bam | pi2 | 6 |
| bat | pi3 | 6 |
The whole setup should be automated and reproducible.
What I’ve done so far
I’ve tried to avoid using an overlay network because of the added complexity and because I figure VLANs should be capable of providing what I need.
I have managed to achieve the desired setup by creating a dedicated network interface on the host and then attaching it to each container:
ubuntu@pi2$ sudo ip link add link eth0 lxdvlan5 type lan id 5 gvrp on # gvrp so that pi advertises the new vlan id to the switch
ubuntu@pi2$ sudo link set dev lxdvlan5 up
ubuntu@pi2$ lxc init --target pi2 ubuntu:20.04 vlantestubuntu
ubuntu@pi2$ lxc config device add vlantestubuntu lxdvlan5 nic nictype=physical parent=lxdvlan5 name=eth0
ubuntu@pi2$ lxc start vlantestubuntu
This works:
- Each container created in this way will have an
eth0that is in the correct VLAN - The host does not see these interfaces while the containers are running
- GVRP advertisements are made to the switch, thus containers on different Pis can talk to each other on VLAN 5
The problem
The above has the slight wrinkle that it requires dedicated ip commands to be run on the host after the container has been created. It thus can’t be automated using lxc on a single cluster node alone.
I have looked at the networking documentation for lxd at https://lxd.readthedocs.io/en/latest/networks/ but have found them to be quite sparse.
Is it possible to achieve what I’m after using lxd alone and if so, can anyone give me some pointers on how to do it?
n.b: I am hoping to avoid using an overlay network if possible because of the added complexity and the fact that VLANs provide enough isolation by themselves.
).
