I’m running OPNsense as an IncusOS VM. I have four network interfaces which are pertinent to this post, which are connected to: the WAN (uplink), the server segment of the network (serverside), the personal devices segment of the network (homeside), and the management interface (management).
IncusOS itself is set to only get addresses on serverside (through which it connects to OPNsense via dhcp/slaac, which is how its default route is configured) and management (which it has a static IP on). This does work; I wasn’t able to get IncusOS to start up for the first time without a network connection, but temporarily making it use DHCP on the uplink on the install seed, then rewriting the network policy after it had connected to check for updates at least once seems to be fine as far as making sure IncusOS can start up later.
But I noticed that the numbers on OPNsense were all wrong, and that an alarming number of WAN addresses were being blocked for coming from the server LAN. After some fiddling, I worked out that IncusOS is forwarding packets it receives on other interfaces (including the WAN and home LAN sides, on which it doesn’t have an IP address) to its default route; I was able to access server IP addresses from the home network, and when I moved the default route to be on the management port instead, the flood of incoming WAN packets moved from the server LAN to the management port from the VM’s point of view. This means it’s routing packets on its own, and as a result, basically circumventing the OPNsense firewall.
I see that the firewall_rules are only meant for access to IncusOS itself, and that the recommendation is to use network ACLs for controlling what packets are forwarded. But the Incus documentation says that network ACLs are not applicable to non-managed and external (physical) managed networks, and Incus forbids editing the configuration of a non-managed network. What’s going on here?
For the time being, keeping the default route on the management port (where no devices except the IncusOS hypervisor and OPNsense VM exist) and firmly tightening what can be routed by the OPNsense box from the management port has at least prevented the immediate threat to the network. But this is, to say the least, not the desired state of affairs. Have I misconfigured something? Is there a way to limit the forwarding that I’m missing out on? Let me know!
IncusOS network config:
confirmation_timeout: 5m
time:
timezone: UTC
interfaces:
- name: "management"
hwaddr: "..."
addresses:
- ".../24"
roles:
- "management"
- "instances"
- name: "uplink"
hwaddr: "..."
roles:
- "instances"
required_for_online: "no"
- name: "unused"
hwaddr: "..."
roles:
- "instances"
required_for_online: "no"
- name: "serverside"
hwaddr: "..."
addresses:
- "dhcp4"
- "slaac"
required_for_online: "no"
roles:
- "management"
- "cluster"
- "storage"
- "instances"
- name: "homeside"
hwaddr: "..."
roles:
- "instances"
required_for_online: "no"
dns:
domain: "..."
hostname: "..."
OPNsense VM config:
architecture: "x86_64"
name: "..."
ephemeral: false
description: "OPNsense-based router/load balancer"
type: "vm"
config:
"boot.autostart": true
"boot.host_shutdown_action": "stop"
"boot.host_shutdown_timeout": 30
"limits.cpu": "sockets=1,cores=6,threads=1"
"limits.disk.priority": 10
"limits.memory": "8GiB"
"limits.memory.hotplug": false
"limits.memory.oom_priority": -1000
"security.agent.metrics": true
"security.csm": true
"security.guestapi": true
"security.iommu": true
"security.protection.delete": true
"security.secureboot": false
"snapshots.expiry": "30d"
"snapshots.pattern": "{{ creation_date|date:'2006-01-02_15-04-05' }}"
"snapshots.schedule": "@hourly"
"snapshots.schedule.stopped": false
"volatile.uuid": "..."
"volatile.uuid.generation": "..."
devices:
root:
"type": "disk"
"size": "16GiB"
"pool": "local"
"path": "/"
eth0:
"type": "nic"
"nictype": "bridged"
"parent": "uplink"
"hwaddr": "..."
eth1:
"type": "nic"
"nictype": "bridged"
"parent": "homeside"
"hwaddr": "..."
eth2:
"type": "nic"
"nictype": "bridged"
"parent": "serverside"
"hwaddr": "..."
eth3:
"type": "nic"
"nictype": "bridged"
"parent": "management"
"hwaddr": "..."
eth4:
"type": "nic"
"nictype": "bridged"
"parent": "unused"
"hwaddr": "..."