I have a setup working reasonably well with an architecture like this:
The blue box is my Incus server.
It has three Ethernet NICs connecting with two independent modem/routers and with my internal LAN.
An OPNsense Virtual Machine with 4 NICs provides separation/connectivity and a couple of simple containers provide services to the Internet.
My current network setup on server (called lxd for historical reasons 
) is quite simple:
mcon@lxd:~$ cat /etc/network/interfaces
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary (LAN/GREEN) network interface
auto enxa0cec887415e
iface enxa0cec887415e inet static
address 192.168.7.99/24
# The secondary (WAN/RED) network interface
auto enp1s0
iface enp1s0 inet static
address 192.168.1.99/24
gateway 192.168.1.1
OPNsense V.M. setup is quite straightforward:
mcon@lxd:~$ incus config show opnsense2
architecture: x86_64
config:
limits.cpu: 0,1
limits.memory: 2GiB
raw.qemu: |
-cpu host
raw.qemu.conf: |
[device "dev-qemu_rng"]
security.secureboot: "false"
volatile.cloud-init.instance-id: 00264431-d0bc-4bc5-916b-d95a5e951406
volatile.eth0.host_name: mac18e39b06
volatile.eth0.hwaddr: 00:16:3e:43:77:c3
volatile.eth0.last_state.created: "false"
volatile.eth1.host_name: mac94db7f06
volatile.eth1.hwaddr: 00:16:3e:96:cf:27
volatile.eth1.last_state.created: "false"
volatile.eth2.host_name: tap14f0fea6
volatile.eth2.hwaddr: 00:16:3e:7d:c3:53
volatile.eth3.host_name: mace2894b2d
volatile.eth3.hwaddr: 00:16:3e:42:eb:33
volatile.eth3.last_state.created: "false"
volatile.last_state.power: RUNNING
volatile.uuid: 9463442f-b5e1-4d70-ae0b-37c642dc3c87
volatile.uuid.generation: 9463442f-b5e1-4d70-ae0b-37c642dc3c87
volatile.vsock_id: "29"
devices:
eth0:
nictype: macvlan
parent: enxa0cec887415e
type: nic
eth1:
nictype: macvlan
parent: enp1s0
type: nic
eth2:
network: ORANGE
type: nic
eth3:
nictype: macvlan
parent: enxa0cec8b35133
type: nic
root:
path: /
pool: default
size: 4GiB
type: disk
ephemeral: false
profiles: []
stateful: false
description: Dual WAN setup for OPNsense
ORANGE network and the two service containers are standard stuff and, with a bit of OPNsense setup everything seems to work fine.
Problem arose when I wanted to use the WiFi adapter on lxd to provide wireless connectivity.
First I tried to handle the whole subsystem to the V.M.
This would be preferred because OPNsense has provisions to handle WiFi, but it proved difficult because WiFi adapter comes in the same âIOMMU groupâ with other stuff:
IOMMU Group 5:
00:13.0 PCI bridge [0604]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 [8086:5ad8] (rev fb)
00:13.1 PCI bridge [0604]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #2 [8086:5ad9] (rev fb)
00:13.2 PCI bridge [0604]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 [8086:5ada] (rev fb)
00:13.3 PCI bridge [0604]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 [8086:5adb] (rev fb)
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
02:00.0 Network controller [0280]: Intel Corporation Wireless 3165 [8086:3165] (rev 81)
I could handle over both âEthernet controllerâ and ânetwork Controllerâ together but my attempts to do so (essentially changing relevant nictype: macvlan into nictype: physical) failed.
Following @stgraber advice on IRC I tried to setup a bridge between wlan0 and enxa0cec887415e (LAN interface).
Bridge itself has an IP address because I need to be able to access lxd(which is headless) also when Firewall is not running for some reason.
To this end I changed
mcon@lxd:~$ cat /etc/network/interfaces
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The secondary (WAN/RED) network interface
auto enp1s0
iface enp1s0 inet static
address 192.168.1.99/24
gateway 192.168.1.1
#iface enxa0cec887415e inet manual
#iface enxa0cec8b35133 inet manual
#iface wlan0 inet manual
auto wbr0
iface wbr0 inet static
bridge_ports enxa0cec887415e wlan0
address 192.168.7.99/24
and also changed in opnsense2 configuration:
...
eth0:
nictype: macvlan
parent: wbr0
type: nic
...
This did not work at all because I could reach lxd server, but apparently packets were not forwarded to Firewall (i.e.: I could ping/ssh/whatever 192.168.7.99 but 192.168.7.254 was âunreachableâ).
I also had a very strange behavior: incus console lxd:opnsense2 --type=vga would fail with error:
Error: Get "https://192.168.7.99:8443/1.0": Unable to connect to: 192.168.7.99:8443 ([dial tcp 192.168.7.99:8443: connect: connection refused])
opnsense2 V.M. was âRUNNINGâ, but apparently not reachable.
I went back to previous config to be able to write this
Any hint on how to solve the problem (or otherwise work around it)would be very welcome
But i hope my idea helps you out a little bit