[SOLVED] Setting up router container to manage all networking

1

I am trying to provision a router jail that will take care of all of my home networking needs. I am running it inside a jail so that I can ‘save’ on power, but more importantly, it is the same physical box as my workstation so I don’t need to walk (but seriously, it is much quicker to get to).

That said, my workstation is the physical host, a desktop computer with 2 NICs. I have assigned both NICs over to the router jail and can see them in the jail:

# incus config device add router wan nic nictype=physical parent=enp2s0 name=wan
# incus config device rm router wan
# incus config device add router wired nic nictype=physical parent=eno1 name=wired
# incus config device rm router wired

Before I cut over to this setup, I have DHCP setup on both interfaces and they’re both on the same subnet.

In the router container, I have modified the resolv.conf such that it points to my upstream DNS server (eventually, it will be 127.0.0.1, but for now, it is on the same subnet). Once I have that set, it is able to get to the Internet just fine.

The last thing I did on the router was to enable IP forwarding:
sysctl -w net.ipv4.ip_forward=1

Now, back to the physical host, the workstation, it only has incusbr0 which is a 10.x.x.x address and a vethXXX interface. I am able to ping the router which is on the other end of that 10.x.x.x address. I updated the default gateway to the incusbr0 ip address of 10.x.x.x and also tried the ip of the router, but no dice:

route add default 10.x.x.x

I can ping the router just fine, but I cannot ping my upstream router nor can I resolve DNS and get to the Internet from the workstation.

I should also mention that my firewall, nftables, is presently empty, so that doesn’t factor in here.

What am I missing here - do I want to use the default gateway as the incusbr0 interface’s IP address because I want all traffic from the workstation to go through the router container. Or, do I want to use the veth interface? The veth interface has no IP address.

I should also mention that I am monitoring traffic on my present router via tcpdump and I don’t see anything being blocked at that level, so I suspect that I am just missing the default route in the right place.

Lastly, for bonus points, eventually, I will want the workstation to be on the same subnet as the wired interface I assigned over to the router. Would I then want to assign incusbr0 to an IP address in my subnet (and the same for eth0 in the router container)?

Thanks for your help.

3

Hi, it’s me again.
If I understand correctly, you want router container to be your home router for manage all network, like this:

Screenshot_2025_0513_114211
Screenshot_2025_0513_1142111029×1377 85.6 KB

I haven’t done it yet, but I’ve though about it. You only need 1 bridge for this. Passthrough a port to router container as wan port, and attach bridge to it as lan port. About this bridge, if you use linux bridge, you need to point ip in router ct lan range, and assign the other port to it, but if you use ovs bridge, you only need to assign port to it, ovs bridge doesn’t need ip. Now, no physic port for host to connect to network, to solve it, use bridge to create a virtual port, then host use this port to connect to network. About incusbr0, no need to config it, it will use host network. About lan devices, they will communicate with the port assigned to bridge, because the bridge will act as a switch, so need to config them too.
Screenshot_2025_0513_122854
Screenshot_2025_0513_122854939×1462 89.8 KB

1 Like
4

Cool, I recognized the username.

Yes, the diagram above is what I want.

Ok, I think I understand.

  1. incusbr0 will be managed by host so that it can talk with containers, don’t mess with this
  2. create a bridge - either Linux or OVS and that is how the host will get to the Internet

Since I want the host to be on the same subnet as the LAN, then I’d want to create a Linux bridge.

I’ll report back once I figure that out :).

EDIT:
These are the steps I performed, it isn’t working yet, but I think I’m in the right direction:

In router

  1. install bridge-utils
  2. brctl addbr gateway # keep interface names simple
  3. ifconfig gateway up
  4. brctl addif gateway wired
  5. brctl addif gateway veth1

In host

  1. ip link add veth0 type veth peer name veth1
  2. ifconfig veth0 10.30.0.180 netmask 255.255.255.0 up
  3. incus config device add router veth1 nic nictype=physical parent=veth1 name=veth1

I tried assigning an IP to the bridge, but once I did that, networking stopped.

EDIT:
I noticed a few things:

  1. the router container seems to have its /etc/resolv.conf periodically updated automatically even if I change it, it changes back
  2. when adding the default route in the workstation, I need to ensure I use the virtual interface I created earlier, ie. route add default gateway veth0
  3. same goes for the default route in the router, it tends to revert to creating a default route through eth0 which I believe is automatically created by incus.

Since incus is setting up these virtual interfaces automatically, I think it is redundant for me to create additional ones. Rather, I should just configure them the way that I see fit. I believe eth0 on the router container corresponds to vethxxx on the host.

In any case, it still is not working for me. I am unable to ping the router from the host (workstation) and vice versa.

EDIT:
router configuration:
eth0: 10.212.68.151 / 255.0.0.0
gateway (bridge): 192.168.1.181 / 255.255.255.0
wan: 192.168.1.240 / 255.255.255.0 [normally would be assigned by IP, but this is on my internal network]
wired: 192.168.1.241 / 255.255.255.0 [normally would be static, and I’d assign 192.168.1.1]
0.0.0.0 191.168.1.1 0.0.0.0 UG 0 0 0 wan

workstation configuration: (host)
incusbr0: 10.212.68.1/255.0.0.0
veth0: 192.168.1.180 / 255.255.255.0
vethf38740c3: no IP assigned
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 veth0

EDIT:
I am running tcpdump on the router container and I can see that the gateway bridge is seeing LAN traffic.

I figured it out veth1 was down. I can now ping the router and the remainder of my network!!!

1 Like
5

Great, my theory worked.

6

I have been messing around with some issues since I rebooted to test my changes were permanent :(.

I moved configuration into scripts in /etc/local.d on both the host and container to ensure networking devices exist and are configured. The remainder of configuration is in init scripts as would normally be configured.

The router container is setup as:

  1. wired: DHCP from current router
  2. wan: DHCP from current router (originally I had left this disconnected to simplify matters)
  3. gateway (bridge) has wired and veth1 in it, I tried with no IP assigned to the veth1 and with one
  4. eth0 is set dynamically by incus
  5. route -n originally had the wired interface as the default gateway with my current router IP as the gateway, which looks correct to me)
  6. after adding the wan interface, route -n showed it as the default gateway, again that also makes sense.

I still have this device connecting to my existing network and it is able to get a DHCP lease; however, after that, I am unable to ping the router or other devices. I flushed the nftables ruleset, so nftables isn’t the issue.

I configured the router container like my current router which is using static ARP to tie a MAC address to an IP address, so that is the only other thing I can think of that might be the issue. I originally did not have ARP entries for the router itself as it is self, but later added it, also to no avail.

I have tcpdump running on my present router attempting to capture any traffic on its LAN (wired) interface or the interface that it logs to if it doesn’t match the above rules (pflog). I see nothing.

Conversely, I’ve run tcpdump on both wired and wan on the router container and also see nothing, to my surprise.

I’m not sure what is going on now and why it worked before. While I think the only real change is the static ARP, that also doesn’t make sense to me as the culprit.

EDIT:
The workstation (host) is able to ping the current router, but the router container cannot ping the router? The routing table of the workstation (host) is going through veth0 which then goes to veth1 and wired via the gateway bridge. Strange.

The workstation can also get to the Internet just fine.

8

Had a similar idea like this in my mind for my next project on reorganising my home network and separating my subnetworks add a DMZ and other stuff. Haven’t had the time to dive into it so can only try to give some ideas.

It is a bit hard following this. Understand you try to explain in much detail you can. The pics from @catfish are great. Think it would be helpful to draw a new picture like above with network information. This way it might be simpler to follow and help.

I would try to start simple:

 * install required network tools on host
 * install Incus with default settings
 * create router vm with required network tools installed
    * give it a fixed IP from incusbr0 network range 
 * create container c1 with required network tools installed
 * change incusbr0
    * disable nat
    * add a dhcp4 gateway (router ip)
 * reboot the host
 * router should be able to ping c1 and same reverse
 * default gateway for c1 is the router
 * on router container
     * add second interface (direct ISP facing interface)
     * enable IP forwarding
     * enable masquerading
     * add correct default gateway to outside world
  * verify router can see the world
  * container c1 should be able to ping outside world

In theory this should work as I haven’t tested it. Now all Incus container will use the router as their gateway. As the host is aware about incusbr0 chaning the default gateway to the router IP it should use the same path.

As mentioned this is how I would try it in first place, use only IP addresses for now and it might not work at all. Next step would be adding your lan, second NIC to incusbr0 bridge. Correct DNS server and be later configured in the DHCP settings.

9

Let’s debug it step by step.

  1. Test ISP router. Connect a normal device to it, if device get an ip by DHCP and can connect to internet, then ISP router if fine, continue to next. Else debug ISP router.
  2. Test router container’s wan NIC. You passthroughed a NIC to it, and the NIC is connected to ISP router. It should be able to get an ip and connect to internet. If failed, the NIC is malfunction. Else continue to next.
  3. Test router container’s incus NIC. It will work by default. But it add some complicates, router container need to make sure gatway must point to wan NIC. If you use ip r, output should be:
0.0.0.0 dev wan scope link                                                                                                                 
0.0.0.0 dev eth0 scope link

4.Test bridge (br0). Don’t connect it to router container yet. Connect 2 devices to br0, manually assign ip, ping each other. If successed, br0 is fine, remove assigned ip, continue to next. Else, debug bridge.
5. Test connect bridge to ISP router. Your host connects to a virtual NIC which is connected to br0. Now it should be able to get an ip by DHCP and connect to internet. Host can ping router container now. If successed, disconnect bridge to ISP router, continue to next. Else? What could happen?
6. Test router container’s lan NIC. It should be connected to br0 now. Host should get an ip from router container, ping each other. If failed, it should be some misconfig in router container. Else, you can make your router container as you router now.

10

I’m using ovs bridge and not familiar with linux bridge, so I did some research.
I find they are really different. If there is only 1 NIC exist, ovs bridge requires to create a virtual port so pc can connect to it, but linux bridge use a hidden port same name as the bridge.
So, I’m sorry for misleading you. Here is new debugging step:

  1. delete the virtual port. Use br0 for host to connect to network.
  2. repeat everything I posted before. Still need to make sure ISP device working, bridge working, router container working.
12

I see, in step 4, you use router cotainer’s port connected to br0 as wan port. But did it connect to both ISP router and br0? If so, of cause router cotainer has network. That’s why I told you don’t connect router cotainer to bridge, too many factors. Create a container only connects to br0 and test. This time make sure br0 work, if not, debug it. If you can’t find why, post br0 config.

And you don’t need to delete incusbr0, just bring down eth0 will be enough.

13

Hmm, smoke pours out ears.

No, wan port is not in a bridge, only the wired interface is in the bridge.

Let me try to draw this in ascii:
router container:

  • wan: DHCP
  • wired: static IP, set via OS networking conf
  • veth1: pair interface corresponding to veth0, assigned to container
  • gateway: bridge
    • veth1
    • wired

host (workstation):

  • veth0: originally assigned a static IP, but moved to DHCP so that the route and DNS would be updated (in theory)
  • incusbr0 and vethxxx were deleted as I could not find any references in my router container referring to them (other than backup.yaml)

I think the gateway (bridge) works as my DHCP request from veth0 gets to veth1 which I think see on wired. I just don’t see a response to the DHCP request for whatever reason.

14

It’s so hard to understand what you wrote. Can you use excalidraw.com to draw you network topology next time?

Anyway you didn’t delete the virtual port as I told you to, is it veth0? That 's why your bridge failed. Host will use the same name as bridge to connect to network, is it gateway?

Here is the right one. Rectangle is device, ellipse is NIC, diamond is bridge. arrow point from lan port to wan port.

2025-05-15T13-37-06.652Z
2025-05-15T13-37-06.652Z1050×678 171 KB

15

Here is my network:

image
image1083×496 27.6 KB

I followed your schema, but put attempted to visually group devices how I think they are.

  1. bridge contains veth1 and wired ports
  2. host has a single veth0 port
  3. router has wired, wan, veth1, and bridge

I presently have the machine disconnected so I can post this; however.

  1. The host has an empty routing table, if it were able to get a DHCP lease, I would expect it to then have a proper routing table
  2. When the machine is connected, the router container has a routing table where it is routing traffic through the public IP on the wan port.

The incus created interfaces are no longer there as they were by default, but I think that is normally only needed for the default out of the box configuration from which I am clearly deviating.

16

OK, I understand now.
I’ve told you twice, host need to use the NIC same name as bridge to connect to network. So delete veth0 and veth1. Give bridge ip by DHCP or static. Test bridge, and it will work.

18

I have done something similar before. This post describes my solution. Might help: https://discuss.linuxcontainers.org/t/issue-rebooting-container-with-physical-nic-device-passthrough

19

Because linux bridge will create a hidden port same name as the bridge. When bridge NIC get an ip, it’s host using it not the bridgge.

21

I might take a look at it later, but I currently don’t have the time to read this entire post and help. I thought my post could help as it has a full setup which works (used to until that router machine stopped working). Variations of what I did work as well and, if I understood correctly, it’s what you’re trying to achieve.

22

Seems linux bridge is too abstract for you. Now use ovs bridge.
Free the NIC linux bridge using. Install openvswitch-switch. Change to root user or use sudo.
Create a bridge named ovsbr0: ovs-vsctl add-br ovsbr0
The NIC you just freed, let’s call it eno1, add it to ovsbr0: ovs-vsctl add-port ovsbr0 eno1
Create a virtual port for host use, let’s call it host: ovs-vsctl add-port ovsbr0 host -- set Interface host type=internal
Now bring host network up: edit /etc/network/interfaces or whatever manage you network:

auto host
allow-hotplug host
iface host inet dhcp

Restart network: service networking restart

Or systemd-networkd: Edit /etc/systemd/network/host.network

[Match]
Name=host
[Network]
DHCP=true

systemctl restart systemd-networkd
Rest is the same as linux bridge.

24

First of all, I don’t use linux bridge. Second, I don’t use gentoo linux. So, what I can do is search and point you the way.
Gentoo linux bridge docs has a session about Single NIC bridge. It’s how you config the port connected to lan.

I think you don’t need veth1. Here is how you add router container to bridge(gateway):
incus config device add router lan nic name=lan nictype=bridged parent=gateway
And manually bring the port up.

26

In an effort to digest this information, I ran it through claude.ai.

Here is the conversation:

https://claude.ai/share/57fc3682-0678-4195-9d65-90a950b40737

Here are the details.

  • Does this help?
  • Is this accurate?
  • Is this an acceptable practice to help make the results of these conversations more accessible in the future?

As part of my “incus everywhere” approach, I very much plan to execute a version of this in my local offices. I will keep everyone posted as I progress.

Container-Based Router Setup Guide

Core Concept

This guide describes how to create a home network setup where:

  1. A single physical host (workstation) has two NICs
  2. A container called “router” running inside this host functions as the router/firewall for the entire home network
  3. The physical NICs are passed through to the router container
  4. The host itself connects to the network through this container

This approach allows you to use one machine as both your workstation and router/firewall, saving power and hardware costs.

Network Architecture

This setup requires:

  1. One NIC (wan) connects to the ISP/upstream router
  2. Another NIC (wired) connects to the local network devices
  3. The router container manages all traffic and provides DHCP, DNS, etc.
  4. The host itself connects to the router container through a bridge interface

Network Diagram

┌─────────────┐          ┌─────────────┐          ┌─────────────┐
│             │          │             │          │             │
│   Internet  │◄────────►│ ISP Router  │◄────────►│    WAN      │
│             │          │             │          │    (enp2s0) │
└─────────────┘          └─────────────┘          │             │
                                                  │  Container  │
                                                  │  "router"   │
                                                  │             │
┌─────────────┐          ┌─────────────┐          │             │
│             │          │             │          │             │
│ LAN Devices │◄────────►│    LAN      │◄────────►│   Bridge    │
│             │          │   (eno1)    │          │  (gateway)  │
└─────────────┘          └─────────────┘          └─────────────┘
                                                        ▲
                                                        │
                                                        ▼
                                                  ┌─────────────┐
                                                  │             │
                                                  │    Host     │
                                                  │ Workstation │
                                                  │             │
                                                  └─────────────┘

Detailed Implementation Guide

1. Physical Setup

  • Host Hardware: A desktop computer with two physical NICs
    • NIC 1 (enp2s0): Will be passed through to container as WAN interface
    • NIC 2 (eno1): Will be passed through to container as wired/LAN interface
  • Software: Incus (container manager), bridge-utils

2. Creating the Router Container

# Create the router container
incus launch <image> router

# Pass the WAN NIC to the container
incus config device add router wan nic nictype=physical parent=enp2s0 name=wan

# Pass the LAN NIC to the container
incus config device add router wired nic nictype=physical parent=eno1 name=wired

3. Host Network Configuration

Creating a bridge on the host provides the connection point between the host and the router container:

# Install bridge utilities if not already present
apt install bridge-utils  # (or equivalent for your distro)

# Create a bridge
brctl addbr gateway

# Bring the bridge up
ip link set gateway up

# Assign IP to the bridge (in the LAN subnet)
ip addr add 192.168.1.180/24 dev gateway

4. Container Network Configuration

Inside the router container:

# Configure the WAN interface (either DHCP or static)
# For DHCP:
dhclient wan

# Configure the LAN interface with a static IP
ip addr add 192.168.1.1/24 dev wired

# Create a bridge and add the LAN interface to it
brctl addbr gateway
brctl addif gateway wired
ip link set gateway up

# Enable IP forwarding
sysctl -w net.ipv4.ip_forward=1

# Set up NAT (using iptables)
iptables -t nat -A POSTROUTING -o wan -j MASQUERADE
iptables -A FORWARD -i gateway -o wan -j ACCEPT
iptables -A FORWARD -i wan -o gateway -m state --state RELATED,ESTABLISHED -j ACCEPT

5. Setting Up DHCP and DNS on Router Container

# Install dnsmasq
apt install dnsmasq

# Configure dnsmasq to provide DHCP and DNS on the LAN
cat > /etc/dnsmasq.conf << EOF
interface=gateway
dhcp-range=192.168.1.50,192.168.1.150,24h
dhcp-option=option:router,192.168.1.1
dhcp-option=option:dns-server,192.168.1.1
EOF

# Restart dnsmasq
systemctl restart dnsmasq

6. Host Routing Configuration

Configure the host to use the router container as its gateway:

# Add default route via the bridge
ip route add default via 192.168.1.1 dev gateway

# Configure DNS to use the router container
echo "nameserver 192.168.1.1" > /etc/resolv.conf

7. Making Configuration Persistent

Create startup scripts to ensure all settings are applied after reboot:

  • For the host, add these configurations to network startup scripts (depending on distro)
  • For the container, create a startup script that applies all the network settings

Key Technical Insights

  1. The Bridge Interface: The Linux bridge creates a virtual switch. When you assign an IP to the bridge itself, the host can use it as a network interface.

  2. Physical NIC Passthrough: By passing physical NICs to the container, the container gets direct control of the hardware.

  3. Traffic Flow:

    • Internet → WAN NIC → Router Container → Bridge → LAN Devices/Host
    • Host → Bridge → Router Container → WAN NIC → Internet

  4. No Need for Veth Pairs: A common confusion is trying to use veth pairs when they aren’t necessary. The bridge itself provides the connection point between host and container.

Troubleshooting

Common Issues

  1. DHCP Not Working:

    • Ensure dnsmasq is properly configured and running
    • Check that the bridge interfaces are properly connected
    • Verify that firewall rules aren’t blocking DHCP requests

  2. Host Can’t Access Internet:

    • Verify IP forwarding is enabled in the container
    • Check that NAT is properly configured
    • Ensure the default route on the host points to the router container

  3. Container Networking Resets After Reboot:

    • Create proper startup scripts to configure networking
    • For Incus, use hooks to ensure network configuration applies at container start

  4. Bridge Issues:

    • Make sure physical interfaces are properly added to bridges
    • Check that bridges have the correct IP configurations
    • Verify that bridge interfaces are up and running

Benefits of This Setup

  1. Resource Efficiency: Uses a single physical machine for both routing and workstation tasks
  2. Isolation: Network services run in a container, isolating them from the host system
  3. Flexibility: Easy to modify network configuration without affecting the host
  4. Security: Provides an additional layer of isolation for network services
  5. Power Savings: No need for a separate dedicated router device
27

When I go do implement this for real, the host will be debian and the router container will be executed in nixos. Does anyone have an interest in working through this together?