General description
I’m trying to setup an OpenWRT container running inside incus to manage my network. The host server is running Debian 12. It has 4 ethernet ports which inside the server are enp2s0
to enp5s0
and I have a bridge br0
setup for the host to connect on the network. The setup I have with this server is that all physical interfaces are passed through to OpenWRT using nictype=physical
, br0
is connected inside the container for communication and a few other incus managed bridges as well. Below I’ll give more details on the configuration files for this precise build.
In particular, everything is working as expected except for when I reboot the container either from the container itself or by using incus restart openwrt
. This usually fails and the container doesn’t come back up due to some physical interface which was passed through to the container leaving the container with a different name. The container doesn’t restart with an error indication that the nic device doesn’t exist.
Has anynoe encountered this issue before? Has anyone fixed this issue or know a particular cause in my setup? Is it a bug?
Configuration details
The host network setup has a bridge with a static IP address and nothing else.
victoitor@router-server:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
victoitor@router-server:~$ cat /etc/network/interfaces.d/br0
## dhcp ip config file for br0
auto br0
iface br0 inet static
address 10.0.0.10
netmask 255.255.255.0
gateway 10.0.0.1
bridge_ports none
bridge_stp off # disable Spanning Tree Protocol
bridge_waitport 0 # no delay before a port becomes available
bridge_fd 0 # no forwarding delay
Incus has a few bridges setup, but dns and dhcp are turned off. I’ll show the configuration of only one of those bridges as the other ones are identical.
victoitor@router-server:~$ incus network list
+-----------+--------+---------+------+------+-------------+---------+---------+
| NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE |
+-----------+--------+---------+------+------+-------------+---------+---------+
| br0 | bridge | NO | | | | 3 | |
+-----------+--------+---------+------+------+-------------+---------+---------+
| dmz | bridge | YES | none | none | | 2 | CREATED |
+-----------+--------+---------+------+------+-------------+---------+---------+
| iot | bridge | YES | none | none | | 2 | CREATED |
+-----------+--------+---------+------+------+-------------+---------+---------+
| services | bridge | YES | none | none | | 2 | CREATED |
+-----------+--------+---------+------+------+-------------+---------+---------+
| untrusted | bridge | YES | none | none | | 2 | CREATED |
+-----------+--------+---------+------+------+-------------+---------+---------+
victoitor@router-server:~$ incus network show iot
config:
dns.mode: none
ipv4.address: none
ipv4.dhcp: "false"
ipv4.firewall: "false"
ipv6.address: none
ipv6.dhcp: "false"
ipv6.firewall: "false"
description: ""
name: iot
type: bridge
used_by:
- /1.0/instances/openwrt
- /1.0/profiles/router
managed: true
status: Created
locations:
- none
This is the container configuration and the router profile which it uses.
victoitor@router-server:~$ incus config show openwrt
architecture: x86_64
config:
image.architecture: amd64
image.description: Openwrt 23.05 amd64 (20240115_11:57)
image.os: Openwrt
image.release: "23.05"
image.serial: "20240115_11:57"
image.type: squashfs
image.variant: default
volatile.base_image: f17085e23771357cdf5e06d99c418707dfdeb0c40a50a97602f2ad7d35ef6d86
volatile.cloud-init.instance-id: 6463fddd-1c48-4ebd-a286-c1be00c19e1b
volatile.eth0.host_name: enp2s0
volatile.eth0.last_state.created: "false"
volatile.eth0.last_state.hwaddr: 60:be:b4:02:59:1c
volatile.eth0.last_state.mtu: "1500"
volatile.eth1.host_name: enp3s0
volatile.eth1.last_state.created: "false"
volatile.eth1.last_state.hwaddr: 60:be:b4:02:59:1d
volatile.eth1.last_state.mtu: "1500"
volatile.eth2.host_name: enp4s0
volatile.eth2.last_state.created: "false"
volatile.eth2.last_state.hwaddr: 60:be:b4:02:59:1e
volatile.eth2.last_state.mtu: "1500"
volatile.eth3-wan.host_name: enp5s0
volatile.eth3-wan.last_state.created: "false"
volatile.eth3-wan.last_state.hwaddr: 60:be:b4:02:59:1f
volatile.eth3-wan.last_state.mtu: "1500"
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
volatile.local-dmz.host_name: veth56b86216
volatile.local-dmz.hwaddr: 00:16:3e:95:05:01
volatile.local-iot.host_name: veth9062c426
volatile.local-iot.hwaddr: 00:16:3e:71:5d:f4
volatile.local-lan.host_name: veth46767acc
volatile.local-lan.hwaddr: 00:16:3e:4b:a4:cb
volatile.local-services.host_name: veth318a27d2
volatile.local-services.hwaddr: 00:16:3e:39:54:b9
volatile.local-untrusted.host_name: veth6ea55423
volatile.local-untrusted.hwaddr: 00:16:3e:f7:cb:2e
volatile.uuid: a3099060-8bf3-4a6e-8e30-f9d841c3d478
volatile.uuid.generation: a3099060-8bf3-4a6e-8e30-f9d841c3d478
devices: {}
ephemeral: false
profiles:
- router
stateful: false
description: ""
victoitor@router-server:~$ incus profile show router
config:
boot.autostart: "true"
boot.autostart.priority: "10"
description: Default router profile
devices:
eth0:
name: eth0
nictype: physical
parent: enp2s0
type: nic
eth1:
name: eth1
nictype: physical
parent: enp3s0
type: nic
eth2:
name: eth2
nictype: physical
parent: enp4s0
type: nic
eth3-wan:
name: eth3-wan
nictype: physical
parent: enp5s0
type: nic
local-dmz:
name: local-dmz
network: dmz
type: nic
local-iot:
name: local-iot
network: iot
type: nic
local-lan:
name: local-lan
nictype: bridged
parent: br0
type: nic
local-services:
name: local-services
network: services
type: nic
local-untrusted:
name: local-untrusted
network: untrusted
type: nic
root:
path: /
pool: default
type: disk
name: router
used_by:
- /1.0/instances/openwrt
This is how the interfaces look like inside and outside the container.
victoitor@router-server:~$ incus exec openwrt -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-base state DOWN qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-base state UP qlen 1000
link/ether 60:be:b4:02:59:1d brd ff:ff:ff:ff:ff:ff
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-base state DOWN qlen 1000
link/ether 60:be:b4:02:59:1e brd ff:ff:ff:ff:ff:ff
5: eth3-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
inet 192.168.0.7/24 brd 192.168.0.255 scope global eth3-wan
valid_lft forever preferred_lft forever
inet6 fe80::62be:b4ff:fe02:591f/64 scope link
valid_lft forever preferred_lft forever
6: br-dmz: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:95:05:01 brd ff:ff:ff:ff:ff:ff
inet 10.0.4.1/24 brd 10.0.4.255 scope global br-dmz
valid_lft forever preferred_lft forever
inet6 fe80::62be:b4ff:fe02:591c/64 scope link
valid_lft forever preferred_lft forever
7: br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
inet6 fe80::62be:b4ff:fe02:591c/64 scope link
valid_lft forever preferred_lft forever
8: br-base.14@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dmz state UP qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
9: br-guest: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global br-guest
valid_lft forever preferred_lft forever
inet6 fe80::62be:b4ff:fe02:591c/64 scope link
valid_lft forever preferred_lft forever
10: br-base.13@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-guest state UP qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
11: local-dmz@br-iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dmz state UP qlen 1000
link/ether 00:16:3e:95:05:01 brd ff:ff:ff:ff:ff:ff
12: br-iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:71:5d:f4 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.1/24 brd 10.0.2.255 scope global br-iot
valid_lft forever preferred_lft forever
inet6 fe80::62be:b4ff:fe02:591c/64 scope link
valid_lft forever preferred_lft forever
13: local-iot@br-base.12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-iot state UP qlen 1000
link/ether 00:16:3e:71:5d:f4 brd ff:ff:ff:ff:ff:ff
14: br-base.12@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-iot state UP qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
15: local-lan@br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
link/ether 00:16:3e:4b:a4:cb brd ff:ff:ff:ff:ff:ff
16: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:4b:a4:cb brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/24 brd 10.0.0.255 scope global br-lan
valid_lft forever preferred_lft forever
inet6 fe80::62be:b4ff:fe02:591c/64 scope link
valid_lft forever preferred_lft forever
17: local-services@br-base.10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-services state UP qlen 1000
link/ether 00:16:3e:39:54:b9 brd ff:ff:ff:ff:ff:ff
18: br-base.10@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
19: local-untrusted@br-services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-untrusted state UP qlen 1000
link/ether 00:16:3e:f7:cb:2e brd ff:ff:ff:ff:ff:ff
20: br-services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:39:54:b9 brd ff:ff:ff:ff:ff:ff
inet 10.0.5.1/24 brd 10.0.5.255 scope global br-services
valid_lft forever preferred_lft forever
inet6 fe80::62be:b4ff:fe02:591c/64 scope link
valid_lft forever preferred_lft forever
21: br-base.15@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-services state UP qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
22: br-untrusted: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:f7:cb:2e brd ff:ff:ff:ff:ff:ff
inet 10.0.1.1/24 brd 10.0.1.255 scope global br-untrusted
valid_lft forever preferred_lft forever
inet6 fe80::62be:b4ff:fe02:591c/64 scope link
valid_lft forever preferred_lft forever
23: br-base.11@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-untrusted state UP qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
victoitor@router-server:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e2:da:41:4f:38:c4 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.10/24 brd 10.0.0.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::e0da:41ff:fe4f:38c4/64 scope link
valid_lft forever preferred_lft forever
7: dmz: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:c9:b0:7a brd ff:ff:ff:ff:ff:ff
8: iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:2b:1b:e1 brd ff:ff:ff:ff:ff:ff
9: services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:3b:c5:a7 brd ff:ff:ff:ff:ff:ff
10: untrusted: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:09:29:e1 brd ff:ff:ff:ff:ff:ff
12: veth56b86216@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master dmz state UP group default qlen 1000
link/ether 36:63:8e:dc:e2:80 brd ff:ff:ff:ff:ff:ff link-netnsid 0
14: veth9062c426@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master iot state UP group default qlen 1000
link/ether 3e:91:57:4a:da:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
16: veth46767acc@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether 7e:6d:e6:c1:fb:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
18: veth318a27d2@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master services state UP group default qlen 1000
link/ether 22:1f:69:8a:a6:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
20: veth6ea55423@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master untrusted state UP group default qlen 1000
link/ether 7e:a0:e1:cf:f0:4d brd ff:ff:ff:ff:ff:ff link-netnsid 0
As I mentioned, everything works fine until there is an attempt to restart the container, in which case it fails to start. The following was obtained by running incus exec openwrt -- reboot
. But something similar occurs when I try incus restart openwrt
.
victoitor@router-server:~$ incus list
+---------+---------+------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------+---------+------+------+-----------+-----------+
| openwrt | STOPPED | | | CONTAINER | 0 |
+---------+---------+------+------+-----------+-----------+
victoitor@router-server:~$ incus start openwrt
Error: Failed to start device "eth1": Parent device 'enp3s0' doesn't exist
Try `incus info --show-log openwrt` for more info
victoitor@router-server:~$ incus info --show-log openwrt
Name: openwrt
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2024/01/15 17:12 -03
Last Used: 2024/01/16 13:51 -03
Log:
victoitor@router-server:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
3: phys20DyIm: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 60:be:b4:02:59:1d brd ff:ff:ff:ff:ff:ff
altname enp3s0
4: physjGqQYH: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 60:be:b4:02:59:1e brd ff:ff:ff:ff:ff:ff
altname enp4s0
5: physgu2bpE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
altname enp5s0
6: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether e2:da:41:4f:38:c4 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.10/24 brd 10.0.0.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::e0da:41ff:fe4f:38c4/64 scope link
valid_lft forever preferred_lft forever
7: dmz: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:c9:b0:7a brd ff:ff:ff:ff:ff:ff
8: iot: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:2b:1b:e1 brd ff:ff:ff:ff:ff:ff
9: services: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:3b:c5:a7 brd ff:ff:ff:ff:ff:ff
10: untrusted: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:09:29:e1 brd ff:ff:ff:ff:ff:ff
This time the naming change happened to enp3s0
up to enp5s0
, but not to enp2s0
. Sometimes the naming change happens to just one interface (usually always happens to enp5s0
).
Is this a bug or is there something odd with my setup?