I used to create fake bridges with openvswitch (i.e. ovs-vsctl add-br vlan200 trunk0 200) and attach a container to that vlan with nictype bridged, like:
devices:
eth0:
nictype: bridged
parent: vlan200
type: nic
Apparently something has changed between incus 0.4 and 0.5.1, since I can’t do that anymore. After some digging I found that you can add a vlan tag nowadays like :
devices:
eth0:
nictype: bridged
parent: trunk0
type: nic
vlan: “200”
I used to be able to see the fake bridges via “incus network list”, but they don’t show up anymore.
They also don’t show up though in version 0.4, so something weird is going on.
Anyway is it correct that from now on I should use the vlan tag and attach a container to the trunk bridge?
NB: on the latest lxd (5.20) it still works though
NB: if it matters, I am running on Archlinux (Endaevouros)
I don’t think we changed anything, do you see those fake bridges in ip link?
It could be that rather than a change in behavior on the Incus side, it’s a change of behavior on your OVS side where those bridges are not exposed as normal interfaces anymore?
Yes I still do see the bridges with ip link. I doubt that incus sees fake bridges as bridges though, since incus does see the trunk bridge, which is the master bridge so to speak, but not the fake ones. Also when I create another bridge like “ovs-vsctl add-br vlan500” (so not a fake one), incus does see it. So I rather think it’s on the incus side as lxd does sees the fake ones as bridges too.
See my output for incus (removed lxd, so can’t show that atm). In my case I am looking at vlan10, not vlan200 as I started the post with:
[root@obelix ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether a8:a1:59:02:d0:6a brd ff:ff:ff:ff:ff:ff
3: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master ovs-system state DOWN mode DEFAULT group default qlen 1000
link/ether a8:a1:59:02:d0:68 brd ff:ff:ff:ff:ff:ff
4: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:dd:24:e0:1d:cf brd ff:ff:ff:ff:ff:ff
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 3a:a1:b2:2e:6e:7e brd ff:ff:ff:ff:ff:ff
6: vlan103: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether b6:47:50:4c:0e:4f brd ff:ff:ff:ff:ff:ff
7: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether b6:47:50:4c:0e:4f brd ff:ff:ff:ff:ff:ff
8: vlan102: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether b6:47:50:4c:0e:4f brd ff:ff:ff:ff:ff:ff
9: vlan100: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether b6:47:50:4c:0e:4f brd ff:ff:ff:ff:ff:ff
10: trunk0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether a8:a1:59:02:d0:68 brd ff:ff:ff:ff:ff:ff
11: vlan33: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether b6:47:50:4c:0e:4f brd ff:ff:ff:ff:ff:ff
61: veth7aad6de4@if60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether ea:64:a0:43:08:ea brd ff:ff:ff:ff:ff:ff link-netnsid 0
63: vethc116e822@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether ee:02:36:61:f8:68 brd ff:ff:ff:ff:ff:ff link-netnsid 3
65: vethbae35ad0@if64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether 66:af:1b:fe:0d:84 brd ff:ff:ff:ff:ff:ff link-netnsid 2
[root@obelix ~]# incus network list
±-------±---------±--------±-----±-----±------------±--------±------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
±-------±---------±--------±-----±-----±------------±--------±------+
| eno1 | physical | NO | | | | 0 | |
±-------±---------±--------±-----±-----±------------±--------±------+
| enp2s0 | physical | NO | | | | 0 | |
±-------±---------±--------±-----±-----±------------±--------±------+
| trunk0 | bridge | NO | | | | 1 | |
±-------±---------±--------±-----±-----±------------±--------±------+
| wlan0 | physical | NO | | | | 0 | |
±-------±---------±--------±-----±-----±------------±--------±------+
[root@obelix ~]# ovs-vsctl show
58961ee6-4cc2-469a-b44d-32598cc71d65
Bridge trunk0
Port enp2s0
Interface enp2s0
Port vlan102
tag: 102
Interface vlan102
type: internal
Port vethbae35ad0
tag: 10
Interface vethbae35ad0
Port vlan103
tag: 103
Interface vlan103
type: internal
Port trunk0
Interface trunk0
type: internal
Port vethc116e822
Interface vethc116e822
Port veth7aad6de4
tag: 10
Interface veth7aad6de4
Port vlan10
tag: 10
Interface vlan10
type: internal
Port vlan100
tag: 100
Interface vlan100
type: internal
Port vlan33
tag: 33
Interface vlan33
type: internal
Port eno1
Interface eno1
We do actually still process those entries and you’ll see something if you do incus network show vlan200 but because the kernel reports their type as unknown, they get excluded from the list we show users to limit the number of spurious entries there.
So that part is normal. The error you’re getting is because of our move to using OpenVswitch’s client library (libovsdb) rather than CLI which is now no longer considering those fake bridges as suitable and is failing.
I’m looking at tweaking that logic now to restore things.
Hmm, so looks like the CLI is doing quite a bit of magic as according to OpenVswitch’s own database, vlan200 is not a bridge and therefore cannot have ports associated with it.
I’ve spent some time tracking down the logic and DB calls made by the OVS CLI tools and it’s pretty crazy what they do… Basically they treat interfaces of type internal as bridges for the purpose of list-br, whether the interface is an actual bridge or not.
When you do ovs-vsctl addbr vlan200 trunk0 200, you don’t actually create a bridge, instead you create a normal interface named vlan200 which is associated to a port on bridge trunk0 with that port having a VLAN of 200 associated to it.
When you then do ovs-vsctl add-port vlan200 XYZ, the CLI tool tracks down the vlan200 interface, tracks down the associated port, notes the VLAN on it (200) and the bridge it’s attached to (trunk0) and then creates a replica of that interface and port for the XYZ device.
This works fine, but is a massive hack as at the end of the process XYZ ends up as a completely standalone interface & port, like vlan200 itself rather than being tied as a child of vlan200 as would be expected with an actual bridge.
Given that, I don’t feel confident at all with us trying to replicate this as it would need similar logic absolutely anywhere we deal with a bridge/interface/port.
I think that moving forward having folks use the actual bridge as parent and then set their VLAN config through vlan and vlan.tagged is the right way to go as that actually lines up with the internal OVS model (database tables, objects, …) and achieves the same thing but with less placeholder interfaces and ports on the system.
Thanks for digging into it, so basically it’s just easier/better to use the vlan tag then, which kind of makes more sense after all, since you want to connect the container to a certain vlan and not to a bridge
