I am currently trying to configure an Incus setup and would like to know if what I want to achieve is feasible or not (so that I do not waste my time trying… even if I already spent couple of hours trying to make this work !)
So I have Incus on Ubuntu 24.04LTS configured with a network interface using the macvlan driver.
On the host, I also have a second interface (VLAN) with an IP address in VLAN X.
I have a VM instance running in Incus on which I want to allocate another IP address on that same VLAN X.
Thing is, I am only able to ping/reach the IP address on the host system, not the one on the VM.
If I turn down the VLAN interface on the host, I am able to ping/reach the one on the VM.
I am suspecting either a routing issue, APR issue but I first want to make sure this is feasible before trying any other ways around.
After some more further tests, it seems like the VLAN kernel module (8021q) is in “cause”.
When unloading the module, guest starts to ping (of course, host no longer reply)…
You may be able to use macvlan on top of the host’s VLAN interface but macvlan has a tendency to be pretty fragile and can have driver-specific behavior.
If possible, the cleaner approach is to configure a bridge on your host system through your distribution’s network configuration tool. You’re mentioning Ubuntu 24.04 LTS so in that case, that’d be netplan.
What I’d probably do in your scenario, considering a physical network interface named eth0 is:
Define eth0 in netplan but set dhcp4: false, dhcp6: false, accept-ra: false so it doesn’t get any configured addresses
Define a eth0.1234 VLAN interface with eth0 as the link and 1234 as the id, also have dhcp4, dhcp6 and accept-ra disabled on this
Define a br1234 bridge interface which includes eth0.1234, that interface will have your host’s IP configuration on it, whether that’s DHCP or static
In Incus, you can then use a nic device which looks like this to add an instance to your VLAN 1234:
type: nic
nictype: bridged
parent: br1234
name: eth0
You can do some fancier things too by having just a single bridge (br0) which would include eth0 and then has VLAN filtering enabled so that you can instead do this in Incus:
type: nic
nictype: bridged
parent: br0
vlan: 1234
name: eth0
That’s what I do on my production servers, but Netplan doesn’t have full support for this, so I need some additional systemd-networkd units on my system to properly handle that (turn on VLAN filtering and configure the list of valid VLAN IDs on the bridge parent device, eth0 in this scenario)
One thing I didn’t mention and that may be important in what I want to achieve is that there are other VLANs that need to be accessible on the guest (not on the host)
Yeah, so the ideal is definitely to enable systemd vlan filtering on the bridge.
You need to do that through a few extra files in /etc/systemd/networkd as Netplan doesn’t support it natively.
Wow that is some great stuff
So basically, I would have to create one nic device for each VLAN and add them to the VM correct ?
Also, I am not sure of the interaction between your br0 and bond-sw01. Where does the host VLAN X address resides in that scenario ?
On a side note, what’s the best way to sponsor you since you are of great help ?
The netplan config above will work but it will have one annoying issue.
As you’re taking VLAN 2 out of eno1, that VLAN will not be available on br0.
So if you have containers that need access to VLAN2, that won’t work.
You have two solutions to fix this:
Change your physical switch config so VLAN2 is the untagged VLAN for your server, at which point you can change your static IP config to be on the br0 interface instead and you can then set VLAN 1 for any container that needs access to that untagged VLAN.
Create another bridge, say br-vlan2, put your vlan2 device into that, put the IP config on top of that bridge and then have the instances that need access to vlan2 to get a nic bridged to that bridge.
I’d obviously go with 1) if you have control over the physical network as that keeps things very clean
Good catch ! One thing tho, I need the VLAN 2 to be “tagged” in the VM for different reason. It then cannot be untagged.
I’d then have to go with the 2 bridges solution
Ah yeah, I’m not aware of any good way to re-number things sadly. It’d be great to have it be untagged on the host but then effectively configure things for that untagged VLAN (treated as VLAN 1 by Linux) to be mapped to say VLAN 200 in the instance.
I wouldn’t be surprised if Linux had some kind of way to do that but I haven’t yet found it, so if anyone knows how to configured a VLAN-enable bridged to do that kind of translation, let me know and we can look at adding that to Incus, it’d be pretty neat!
