LXD, Bridge and VLAN(s)

LXD
,
1

Hi

I’m trying to get lxd containers to connect to different vlans.
Initially I tried setting the vlan id on the container nic lxc config device override test eth0 vlan=3 but I got Error: Failed to start device "eth0": VLAN filtering is not enabled in parent bridge "br0" so I enabled vlan filtering in the bridge config VLANFiltering=yes this got rid of the error but it doesn’t actually seem to work.
Could someone please tell me what I’m doing wrong/point me in the right direction, many thanks!

Here’s all my systemd-networkd config

/etc/systemd/network/br0.netdev 
[NetDev]
Name=br0
Kind=bridge
  
[Bridge]
DefaultPVID=1
VLANFiltering=yes
/etc/systemd/network/br0.network 
[Match]
Name=br0
  
[Network]
DHCP=ipv4
  
[DHCPv4]
RouteMetric=100
/etc/systemd/network/eno1.network 
[Match]
Name=eno1
  
[Network]
Bridge=br0
2

What does sudo cat /sys/class/net/br0/bridge/vlan_filtering show?

3 
#> sudo cat /sys/class/net/br0/bridge/vlan_filtering
1
4

OK interesting, thats what it should be

Can you try

sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- cat /sys/class/net/br0/bridge/vlan_filtering
5

I’m running LXD natively on Arch linux (not through snapd)

1 Like
6

Strange. So you can see where the error is coming from:

And the function that loads that value from /sys and trims it:

Suggests LXD is getting a different value.

7

Please can you show lxc config show test --expanded?

8 
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Archlinux current amd64 (20220915_09:41)
  image.os: Archlinux
  image.release: current
  image.requirements.secureboot: "false"
  image.serial: "20220915_09:41"
  image.type: squashfs
  image.variant: default
  security.nesting: "true"
  security.secureboot: "false"
  volatile.apply_template: rename
  volatile.base_image: 4c5da19f7c86feb1dda3ff0254470b7fea347cceeb621e7867563dc9ba8b0040
  volatile.cloud-init.instance-id: ccf56c00-44e4-47ac-bed3-3f9520309b71
  volatile.eth0.hwaddr: 00:16:3e:9b:b0:d4
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: 1604b314-ff18-4df3-a533-a2699b3ac218
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
    vlan: "3"
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

#> bridge vlan show
...
veth28a32059      3 PVID Egress Untagged
...
9

So its working now?

I’m a bit confused what you’re saying isn’t working. Can you provide more details please?

10

Apologies perhaps my question is poorly worded,
To be clear vlan_filtering (initially) wan’t enabled which is why I was getting the error so I just enabled that so LXD is actually detecting it correctly. However the container/nic isn’t able to communicate with the tagged network and I’ve already checked the network on VLAN3 on a physical device to make sure it works (and it does) My question is how do I connect a LXD container to a VLAN is there a step/additional configuration I’m missing?

I was following this article but I’m using systemd-networkd instead of manually setting everything up using ip.

11

Right I see. I would start by reviewing your firewall rules (maybe turn them off whilst debugging to simplify).

Then use tcpdump on br0’s parent interface to check the tagging that is egressing from the instance via br0 and onto the parent interface.

See Capture VLAN tags by using tcpdump - Red Hat Customer Portal

12

Firewall’s completely off.
I will report back with the tcpdump results.
So there’s nothing wrong with my config then, right? I mean theoretically this should (have) just work(ed)?

1 Like
13

tcpdump revealed the vlan wasn’t being forwarded to the underlying interface, after some digging around, turns out it was a misconfiguration issue. According to man 5 systemd.network you must configure allowed VLAN id(s) under [BridgeVLAN] for whatever the slave bridge interface is e.g. bond, physical interface or whatever else.

Example 6. Bridge port with VLAN forwarding

           # /etc/systemd/network/25-bridge-slave-interface-1.network
           [Match]
           Name=enp2s0

           [Network]
           Bridge=bridge0

           [BridgeVLAN]
           VLAN=1-32
           PVID=42
           EgressUntagged=42

           [BridgeVLAN]
           VLAN=100-200

           [BridgeVLAN]
           EgressUntagged=300-400

       This overrides the configuration specified in the previous example for the interface "enp2s0", and enables VLAN on that bridge
       port. VLAN IDs 1-32, 42, 100-400 will be allowed. Packets tagged with VLAN IDs 42, 300-400 will be untagged when they leave on
       this interface. Untagged packets which arrive on this interface will be assigned VLAN ID 42.

So here’s a simple and quick config for setting up a bond and a vlan aware bridge on top.

/etc/systemd/network/bond0.netdev 
[NetDev]
Name=bond0
Kind=bond

[Bond]
Mode=802.3ad
TransmitHashPolicy=layer2+3
MIIMonitorSec=1s
LACPTransmitRate=fast
/etc/systemd/network/bond0-slaves.network 
[Match]
MACAddress=<address1> <address2> ... <addressN>

[Network]
Bond=bond0
/etc/systemd/network/br0.netdev 
[NetDev]
Name=br0
Kind=bridge

[Bridge]
VLANFiltering=yes
Priority=0
DefaultPVID=1
/etc/systemd/network/br0-slave.network 
[Match]
Name=bond0

[Network]
Bridge=br0

[BridgeVLAN]
VLAN=<A>-<B>
PVID=1
EgressUntagged=1
/etc/systemd/network/br0.network 
[Match]
Name=br0

[Network]
DHCP=yes

I Should Note that having fixed one problem now I’m facing another, in that, once VLANFiltering is enabled the bridge doesn’t automatically acquire a dhcp lease unless/until something is “plugged into” it for example if I have LXD.service disabled so that I can start it manually after boot the host system doesn’t have a routable IP until I start lxd which “plugs in” its interfaces into the bridge and suddenly the bridge starts working and gets an IP from the DHCP server.
I don’t know if this is a bug, But I’ve done quite a bit of digging around and this doesn’t seem to be documented/intended behaviour :confused:

PS, the following article from Red Hat is a pretty good read for anyone interested.

1 Like
14

Sounds like you need to enable br0 to come up on boot?

15

negative, bridge is enabled and comes up on boot it just refuses to get a dhcp lease on its own (until something gets plugged in) note that this behaviour is only present/observed when VLANFiltering is enabled

1 Like