Issue rebooting container with physical nic device passthrough

Incus
1

General description

I’m trying to setup an OpenWRT container running inside incus to manage my network. The host server is running Debian 12. It has 4 ethernet ports which inside the server are enp2s0 to enp5s0 and I have a bridge br0 setup for the host to connect on the network. The setup I have with this server is that all physical interfaces are passed through to OpenWRT using nictype=physical, br0 is connected inside the container for communication and a few other incus managed bridges as well. Below I’ll give more details on the configuration files for this precise build.

In particular, everything is working as expected except for when I reboot the container either from the container itself or by using incus restart openwrt. This usually fails and the container doesn’t come back up due to some physical interface which was passed through to the container leaving the container with a different name. The container doesn’t restart with an error indication that the nic device doesn’t exist.

Has anynoe encountered this issue before? Has anyone fixed this issue or know a particular cause in my setup? Is it a bug?

Configuration details

The host network setup has a bridge with a static IP address and nothing else.

victoitor@router-server:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback
victoitor@router-server:~$ cat /etc/network/interfaces.d/br0 
## dhcp ip config file for br0
auto br0
iface br0 inet static
	address			10.0.0.10
	netmask			255.255.255.0
	gateway			10.0.0.1
	bridge_ports    none
	bridge_stp		off			# disable Spanning Tree Protocol
	bridge_waitport	0			# no delay before a port becomes available
	bridge_fd		0			# no forwarding delay

Incus has a few bridges setup, but dns and dhcp are turned off. I’ll show the configuration of only one of those bridges as the other ones are identical.

victoitor@router-server:~$ incus network list
+-----------+--------+---------+------+------+-------------+---------+---------+
|   NAME    |  TYPE  | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |  STATE  |
+-----------+--------+---------+------+------+-------------+---------+---------+
| br0       | bridge | NO      |      |      |             | 3       |         |
+-----------+--------+---------+------+------+-------------+---------+---------+
| dmz       | bridge | YES     | none | none |             | 2       | CREATED |
+-----------+--------+---------+------+------+-------------+---------+---------+
| iot       | bridge | YES     | none | none |             | 2       | CREATED |
+-----------+--------+---------+------+------+-------------+---------+---------+
| services  | bridge | YES     | none | none |             | 2       | CREATED |
+-----------+--------+---------+------+------+-------------+---------+---------+
| untrusted | bridge | YES     | none | none |             | 2       | CREATED |
+-----------+--------+---------+------+------+-------------+---------+---------+
victoitor@router-server:~$ incus network show iot
config:
  dns.mode: none
  ipv4.address: none
  ipv4.dhcp: "false"
  ipv4.firewall: "false"
  ipv6.address: none
  ipv6.dhcp: "false"
  ipv6.firewall: "false"
description: ""
name: iot
type: bridge
used_by:
- /1.0/instances/openwrt
- /1.0/profiles/router
managed: true
status: Created
locations:
- none

This is the container configuration and the router profile which it uses.

victoitor@router-server:~$ incus config show openwrt
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Openwrt 23.05 amd64 (20240115_11:57)
  image.os: Openwrt
  image.release: "23.05"
  image.serial: "20240115_11:57"
  image.type: squashfs
  image.variant: default
  volatile.base_image: f17085e23771357cdf5e06d99c418707dfdeb0c40a50a97602f2ad7d35ef6d86
  volatile.cloud-init.instance-id: 6463fddd-1c48-4ebd-a286-c1be00c19e1b
  volatile.eth0.host_name: enp2s0
  volatile.eth0.last_state.created: "false"
  volatile.eth0.last_state.hwaddr: 60:be:b4:02:59:1c
  volatile.eth0.last_state.mtu: "1500"
  volatile.eth1.host_name: enp3s0
  volatile.eth1.last_state.created: "false"
  volatile.eth1.last_state.hwaddr: 60:be:b4:02:59:1d
  volatile.eth1.last_state.mtu: "1500"
  volatile.eth2.host_name: enp4s0
  volatile.eth2.last_state.created: "false"
  volatile.eth2.last_state.hwaddr: 60:be:b4:02:59:1e
  volatile.eth2.last_state.mtu: "1500"
  volatile.eth3-wan.host_name: enp5s0
  volatile.eth3-wan.last_state.created: "false"
  volatile.eth3-wan.last_state.hwaddr: 60:be:b4:02:59:1f
  volatile.eth3-wan.last_state.mtu: "1500"
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.local-dmz.host_name: veth56b86216
  volatile.local-dmz.hwaddr: 00:16:3e:95:05:01
  volatile.local-iot.host_name: veth9062c426
  volatile.local-iot.hwaddr: 00:16:3e:71:5d:f4
  volatile.local-lan.host_name: veth46767acc
  volatile.local-lan.hwaddr: 00:16:3e:4b:a4:cb
  volatile.local-services.host_name: veth318a27d2
  volatile.local-services.hwaddr: 00:16:3e:39:54:b9
  volatile.local-untrusted.host_name: veth6ea55423
  volatile.local-untrusted.hwaddr: 00:16:3e:f7:cb:2e
  volatile.uuid: a3099060-8bf3-4a6e-8e30-f9d841c3d478
  volatile.uuid.generation: a3099060-8bf3-4a6e-8e30-f9d841c3d478
devices: {}
ephemeral: false
profiles:
- router
stateful: false
description: ""
victoitor@router-server:~$ incus profile show router
config:
  boot.autostart: "true"
  boot.autostart.priority: "10"
description: Default router profile
devices:
  eth0:
    name: eth0
    nictype: physical
    parent: enp2s0
    type: nic
  eth1:
    name: eth1
    nictype: physical
    parent: enp3s0
    type: nic
  eth2:
    name: eth2
    nictype: physical
    parent: enp4s0
    type: nic
  eth3-wan:
    name: eth3-wan
    nictype: physical
    parent: enp5s0
    type: nic
  local-dmz:
    name: local-dmz
    network: dmz
    type: nic
  local-iot:
    name: local-iot
    network: iot
    type: nic
  local-lan:
    name: local-lan
    nictype: bridged
    parent: br0
    type: nic
  local-services:
    name: local-services
    network: services
    type: nic
  local-untrusted:
    name: local-untrusted
    network: untrusted
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: router
used_by:
- /1.0/instances/openwrt

This is how the interfaces look like inside and outside the container.

victoitor@router-server:~$ incus exec openwrt -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-base state DOWN qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-base state UP qlen 1000
    link/ether 60:be:b4:02:59:1d brd ff:ff:ff:ff:ff:ff
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-base state DOWN qlen 1000
    link/ether 60:be:b4:02:59:1e brd ff:ff:ff:ff:ff:ff
5: eth3-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.7/24 brd 192.168.0.255 scope global eth3-wan
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591f/64 scope link 
       valid_lft forever preferred_lft forever
6: br-dmz: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:95:05:01 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.1/24 brd 10.0.4.255 scope global br-dmz
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
7: br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
8: br-base.14@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dmz state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
9: br-guest: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global br-guest
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
10: br-base.13@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-guest state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
11: local-dmz@br-iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dmz state UP qlen 1000
    link/ether 00:16:3e:95:05:01 brd ff:ff:ff:ff:ff:ff
12: br-iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:71:5d:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.1/24 brd 10.0.2.255 scope global br-iot
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
13: local-iot@br-base.12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-iot state UP qlen 1000
    link/ether 00:16:3e:71:5d:f4 brd ff:ff:ff:ff:ff:ff
14: br-base.12@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-iot state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
15: local-lan@br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 00:16:3e:4b:a4:cb brd ff:ff:ff:ff:ff:ff
16: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:4b:a4:cb brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
17: local-services@br-base.10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-services state UP qlen 1000
    link/ether 00:16:3e:39:54:b9 brd ff:ff:ff:ff:ff:ff
18: br-base.10@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
19: local-untrusted@br-services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-untrusted state UP qlen 1000
    link/ether 00:16:3e:f7:cb:2e brd ff:ff:ff:ff:ff:ff
20: br-services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:39:54:b9 brd ff:ff:ff:ff:ff:ff
    inet 10.0.5.1/24 brd 10.0.5.255 scope global br-services
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
21: br-base.15@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-services state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
22: br-untrusted: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:f7:cb:2e brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 brd 10.0.1.255 scope global br-untrusted
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
23: br-base.11@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-untrusted state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
victoitor@router-server:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e2:da:41:4f:38:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::e0da:41ff:fe4f:38c4/64 scope link 
       valid_lft forever preferred_lft forever
7: dmz: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:c9:b0:7a brd ff:ff:ff:ff:ff:ff
8: iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:2b:1b:e1 brd ff:ff:ff:ff:ff:ff
9: services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:3b:c5:a7 brd ff:ff:ff:ff:ff:ff
10: untrusted: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:09:29:e1 brd ff:ff:ff:ff:ff:ff
12: veth56b86216@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master dmz state UP group default qlen 1000
    link/ether 36:63:8e:dc:e2:80 brd ff:ff:ff:ff:ff:ff link-netnsid 0
14: veth9062c426@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master iot state UP group default qlen 1000
    link/ether 3e:91:57:4a:da:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0
16: veth46767acc@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 7e:6d:e6:c1:fb:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
18: veth318a27d2@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master services state UP group default qlen 1000
    link/ether 22:1f:69:8a:a6:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
20: veth6ea55423@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master untrusted state UP group default qlen 1000
    link/ether 7e:a0:e1:cf:f0:4d brd ff:ff:ff:ff:ff:ff link-netnsid 0

As I mentioned, everything works fine until there is an attempt to restart the container, in which case it fails to start. The following was obtained by running incus exec openwrt -- reboot. But something similar occurs when I try incus restart openwrt.

victoitor@router-server:~$ incus list
+---------+---------+------+------+-----------+-----------+
|  NAME   |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+---------+---------+------+------+-----------+-----------+
| openwrt | STOPPED |      |      | CONTAINER | 0         |
+---------+---------+------+------+-----------+-----------+
victoitor@router-server:~$ incus start openwrt
Error: Failed to start device "eth1": Parent device 'enp3s0' doesn't exist
Try `incus info --show-log openwrt` for more info
victoitor@router-server:~$ incus info --show-log openwrt
Name: openwrt
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2024/01/15 17:12 -03
Last Used: 2024/01/16 13:51 -03

Log:


victoitor@router-server:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
3: phys20DyIm: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1d brd ff:ff:ff:ff:ff:ff
    altname enp3s0
4: physjGqQYH: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1e brd ff:ff:ff:ff:ff:ff
    altname enp4s0
5: physgu2bpE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
    altname enp5s0
6: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether e2:da:41:4f:38:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::e0da:41ff:fe4f:38c4/64 scope link 
       valid_lft forever preferred_lft forever
7: dmz: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:c9:b0:7a brd ff:ff:ff:ff:ff:ff
8: iot: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:2b:1b:e1 brd ff:ff:ff:ff:ff:ff
9: services: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:3b:c5:a7 brd ff:ff:ff:ff:ff:ff
10: untrusted: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:09:29:e1 brd ff:ff:ff:ff:ff:ff

This time the naming change happened to enp3s0 up to enp5s0, but not to enp2s0. Sometimes the naming change happens to just one interface (usually always happens to enp5s0).

Is this a bug or is there something odd with my setup?

[SOLVED] Setting up router container to manage all networking
2

Interfaces changing names are always a bit tricky to handle unfortunately.

To start, I’d recommend you try to reproduce this behavior with a basic Ubuntu or Debian image, see if it happens outside of openwrt.

If it doesn’t, then it’d indicate that the issue lies with OpenWRT renaming the interfaces inside of the container.

There’s a basic assumption with passhtrough that the interface will be taken from the host using the name set as parent, then moved into the container with the name set as name and that when the container shuts down, the interface can be taken again from the container using the name from name and restored to the host with the name from parent.

If the interface has been renamed by the container, then that last step can’t work, which then causes a situation like the one you’re describing.

3

Tested on both images:debian/12 and images:ubuntu/22.04 and both exhibited the same bug. It’s interesting to note it’s not always the same network interface (there is some randomness involved) and I have yet to see it happen to enp2s0. Maybe to be able to reproduce it, you need to have more network cards. It doesn’t happen on the first one, so it might not happen if there is only one.

4

Are you getting any error in /var/log/incus/incusd.log?

5 
root@router-server:/var/log/incus# ls
d1  incusd.log	incusd.log.1  liked-foal  openwrt  precious-rhino  u1
root@router-server:/var/log/incus# cat incusd.log
root@router-server:/var/log/incus# cat incusd.log.1
time="2024-01-15T14:26:10-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-15T14:26:10-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T01:54:22-03:00" level=warning msg="Failed to retrieve network information via netlink" instance=openwrt instanceType=container pid=25882 project=default
time="2024-01-16T01:54:24-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/25871/fd/4 1165 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T01:55:45-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T01:55:45-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T01:55:50-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": Unable to connect to: images.linuxcontainers.org:443 ([dial tcp 45.45.148.8:443: connect: no route to host dial tcp [2602:fc62:a:1::8]:443: connect: network is unreachable])"
time="2024-01-16T01:55:50-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": Unable to connect to: images.linuxcontainers.org:443 ([dial tcp 45.45.148.8:443: connect: no route to host dial tcp [2602:fc62:a:1::8]:443: connect: network is unreachable])"
time="2024-01-16T02:13:35-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1294/fd/4 812 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T02:14:01-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T02:14:01-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T02:14:06-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": Unable to connect to: images.linuxcontainers.org:443 ([dial tcp 45.45.148.8:443: connect: no route to host dial tcp [2602:fc62:a:1::8]:443: connect: network is unreachable])"
time="2024-01-16T02:14:06-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": Unable to connect to: images.linuxcontainers.org:443 ([dial tcp 45.45.148.8:443: connect: no route to host dial tcp [2602:fc62:a:1::8]:443: connect: network is unreachable])"
time="2024-01-16T02:22:23-03:00" level=error msg="Failed updating images" err="context canceled"
time="2024-01-16T02:27:29-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1294/fd/4 812 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T02:27:55-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T02:27:55-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T02:29:39-03:00" level=error msg="Failed updating instance types" err="context canceled"
time="2024-01-16T02:29:39-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": context canceled"
time="2024-01-16T02:29:45-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1296/fd/4 813 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T02:30:05-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T02:30:05-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T02:30:07-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"
time="2024-01-16T02:30:07-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"
time="2024-01-16T02:30:07-03:00" level=warning msg="Failed auto start instance attempt" attempt=1 err="Failed to start device \"local-lan\": Parent device \"br0\" doesn't exist" instance=openwrt maxAttempts=3 project=default
time="2024-01-16T02:30:12-03:00" level=warning msg="Failed auto start instance attempt" attempt=2 err="Failed to start device \"local-lan\": Parent device \"br0\" doesn't exist" instance=openwrt maxAttempts=3 project=default
time="2024-01-16T02:30:17-03:00" level=warning msg="Failed auto start instance attempt" attempt=3 err="Failed to start device \"local-lan\": Parent device \"br0\" doesn't exist" instance=openwrt maxAttempts=3 project=default
time="2024-01-16T02:30:17-03:00" level=error msg="Failed to auto start instance" err="Failed to start device \"local-lan\": Parent device \"br0\" doesn't exist" instance=openwrt project=default
time="2024-01-16T02:35:38-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T02:35:38-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T02:36:51-03:00" level=error msg="Failed to update the image" err="Failed getting remote image info: Failed getting image: The requested image couldn't be found" fingerprint=f17085e23771357cdf5e06d99c418707dfdeb0c40a50a97602f2ad7d35ef6d86
time="2024-01-16T02:40:32-03:00" level=error msg="Failed updating instance types" err="context canceled"
time="2024-01-16T02:40:32-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": context canceled"
time="2024-01-16T02:40:38-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1298/fd/4 814 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T02:40:38-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1298/fd/4 814 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T02:40:59-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T02:40:59-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T02:48:11-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1171/fd/4 709 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T02:48:39-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T02:48:39-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T02:50:44-03:00" level=error msg="Failed updating instance types" err="context canceled"
time="2024-01-16T02:50:44-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": context canceled"
time="2024-01-16T02:50:50-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1294/fd/4 811 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T02:51:10-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T02:51:10-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T13:46:37-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1171/fd/4 710 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T13:49:51-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T13:49:51-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T13:50:31-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1185/fd/4 707 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T13:51:37-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T13:51:37-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T15:16:00-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1189/fd/4 709 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T15:16:00-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1189/fd/4 709 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T15:16:00-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1189/fd/4 709 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T15:23:10-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T15:23:10-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T18:37:46-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T18:37:46-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T18:40:40-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1190/fd/4 708 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T18:41:56-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T18:41:56-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T18:42:18-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:56333->8.8.8.8:53: i/o timeout"
time="2024-01-16T18:42:18-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:56333->8.8.8.8:53: i/o timeout"
time="2024-01-16T18:44:57-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1264/fd/4 706 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T18:45:46-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T18:45:46-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T18:46:08-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:41824->8.8.8.8:53: i/o timeout"
time="2024-01-16T18:46:08-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:41824->8.8.8.8:53: i/o timeout"
time="2024-01-16T18:47:07-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 709 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-16T18:47:08-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 709 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-16T18:47:08-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 709 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-16T18:47:50-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T18:47:50-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T18:48:12-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:57937->8.8.8.8:53: i/o timeout"
time="2024-01-16T18:48:12-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:57937->8.8.8.8:53: i/o timeout"
time="2024-01-16T18:48:57-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1257/fd/4 707 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=u1 instanceType=container project=default
time="2024-01-16T18:48:57-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1257/fd/4 707 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=u1 instanceType=container project=default
time="2024-01-16T18:50:42-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T18:50:42-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-16T18:50:59-03:00" level=error msg="Failed updating instance types" err="context canceled"
time="2024-01-16T18:50:59-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/azure.yaml\": context canceled"
time="2024-01-16T18:51:05-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1187/fd/4 706 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T18:51:05-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1187/fd/4 706 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T18:51:05-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1187/fd/4 706 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-16T18:51:34-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-16T18:51:34-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
6

Some of those errors might be when I was testing the issue on ubuntu and debian. I had to stop the openwrt container and lost connectivity.

7

Right so that confirms the theory that it’s the interface having been renamed that causes issues.

For physical nic, it’s imperative that the name set in the name key in the Incus config matches the name inside of the container at the time the container is stopped. Otherwise there is no way for Incus to locate the NIC and rename it back to its correct name.

That’s what all those Device XYZ does not exist errors are reporting here.

8

I’m trying my best to figure out what exactly is going on here. It is odd that this fails on ubuntu and on debian as well.

From my understanding , I found forknet detach to be a call to RunDetach. I believe the call which errored out is err := addr.Flush(). That line is calling this function in addr.go which is essentially calling ip addr flush dev eth1. It seems like forknet is supposed to run this command inside the container’s network namespace, from what I understood.

Some things are really hard to understand here. First, I put out above the result of running ip a on the container and on the host and I’ll reproduce it here.

root@router-server:/sys/class/net# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e2:da:41:4f:38:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::e0da:41ff:fe4f:38c4/64 scope link 
       valid_lft forever preferred_lft forever
7: services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:3b:c5:a7 brd ff:ff:ff:ff:ff:ff
8: untrusted: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:09:29:e1 brd ff:ff:ff:ff:ff:ff
9: dmz: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:c9:b0:7a brd ff:ff:ff:ff:ff:ff
10: iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:2b:1b:e1 brd ff:ff:ff:ff:ff:ff
12: veth9ca63e79@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master dmz state UP group default qlen 1000
    link/ether e6:d0:a6:45:ff:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0
14: vethc273b1aa@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master iot state UP group default qlen 1000
    link/ether b6:79:77:b6:5a:33 brd ff:ff:ff:ff:ff:ff link-netnsid 0
16: vethdbd492f6@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 2e:29:10:e5:43:54 brd ff:ff:ff:ff:ff:ff link-netnsid 0
18: veth5c834edf@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master services state UP group default qlen 1000
    link/ether b2:d9:aa:9a:c6:1e brd ff:ff:ff:ff:ff:ff link-netnsid 0
20: veth319c6919@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master untrusted state UP group default qlen 1000
    link/ether 42:3e:02:3a:2b:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0
root@router-server:/sys/class/net# incus exec openwrt -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-base state DOWN qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-base state UP qlen 1000
    link/ether 60:be:b4:02:59:1d brd ff:ff:ff:ff:ff:ff
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-base state DOWN qlen 1000
    link/ether 60:be:b4:02:59:1e brd ff:ff:ff:ff:ff:ff
5: eth3-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.7/24 brd 192.168.0.255 scope global eth3-wan
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591f/64 scope link 
       valid_lft forever preferred_lft forever
6: br-dmz: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:95:05:01 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.1/24 brd 10.0.4.255 scope global br-dmz
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
7: br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
8: br-base.14@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dmz state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
9: br-guest: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global br-guest
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
10: br-base.13@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-guest state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
11: local-dmz@br-iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dmz state UP qlen 1000
    link/ether 00:16:3e:95:05:01 brd ff:ff:ff:ff:ff:ff
12: br-iot: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:71:5d:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.1/24 brd 10.0.2.255 scope global br-iot
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
13: local-iot@br-base.12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-iot state UP qlen 1000
    link/ether 00:16:3e:71:5d:f4 brd ff:ff:ff:ff:ff:ff
14: br-base.12@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-iot state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
15: local-lan@br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 00:16:3e:4b:a4:cb brd ff:ff:ff:ff:ff:ff
16: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:4b:a4:cb brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
17: local-services@br-base.10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-services state UP qlen 1000
    link/ether 00:16:3e:39:54:b9 brd ff:ff:ff:ff:ff:ff
18: br-base.10@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
19: local-untrusted@br-services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-untrusted state UP qlen 1000
    link/ether 00:16:3e:f7:cb:2e brd ff:ff:ff:ff:ff:ff
20: br-services: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:39:54:b9 brd ff:ff:ff:ff:ff:ff
    inet 10.0.5.1/24 brd 10.0.5.255 scope global br-services
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
21: br-base.15@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-services state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
22: br-untrusted: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:f7:cb:2e brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 brd 10.0.1.255 scope global br-untrusted
       valid_lft forever preferred_lft forever
    inet6 fe80::62be:b4ff:fe02:591c/64 scope link 
       valid_lft forever preferred_lft forever
23: br-base.11@br-base: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-untrusted state UP qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff

I’m not sure in which namespace that command was run, but the interface is not renamed inside the container.

The other thing I find particularly odd concerns the code for RunDetach which I’ll reproduce the lines preceding the errored call here.

func (c *cmdForknet) RunDetach(cmd *cobra.Command, args []string) error {
	daemonPID := args[1]
	ifName := args[2]
	hostName := args[3]

	if daemonPID == "" {
		return fmt.Errorf("Daemon PID argument is required")
	}

	if ifName == "" {
		return fmt.Errorf("ifname argument is required")
	}

	if hostName == "" {
		return fmt.Errorf("hostname argument is required")
	}

	// Check if the interface exists.
	if !util.PathExists(fmt.Sprintf("/sys/class/net/%s", ifName)) {
		return fmt.Errorf("Couldn't restore host interface %q as container interface %q couldn't be found", hostName, ifName)
	}

	// Remove all IP addresses from interface before moving to parent netns.
	// This is to avoid any container address config leaking into host.
	addr := &ip.Addr{
		DevName: ifName,
	}

	err := addr.Flush()
	if err != nil {
		return err
	}

Going backwards from the errored line, the previous line is essentially just defining the Addr struct present on the addr.go file to call addr.Flush(). The main thing we can take from that line is that Flush() is called using the ifName variable. Now, the test right before the struct is defined becomes quite interesting as well. It basically searches for the ifName inside /sys/class/net/ and finds it. If it has just found it then, why can’t it find it on the next call?

For reference, I’m listing the contents of that directory both on the host and on the container.

root@router-server:/sys/class/net# ls -l /sys/class/net/
total 0
lrwxrwxrwx 1 root root 0 jan 16 18:51 br0 -> ../../devices/virtual/net/br0
lrwxrwxrwx 1 root root 0 jan 16 18:51 dmz -> ../../devices/virtual/net/dmz
lrwxrwxrwx 1 root root 0 jan 16 18:51 iot -> ../../devices/virtual/net/iot
lrwxrwxrwx 1 root root 0 jan 16 18:51 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx 1 root root 0 jan 16 18:51 services -> ../../devices/virtual/net/services
lrwxrwxrwx 1 root root 0 jan 16 18:51 untrusted -> ../../devices/virtual/net/untrusted
lrwxrwxrwx 1 root root 0 jan 16 18:51 veth319c6919 -> ../../devices/virtual/net/veth319c6919
lrwxrwxrwx 1 root root 0 jan 16 18:51 veth5c834edf -> ../../devices/virtual/net/veth5c834edf
lrwxrwxrwx 1 root root 0 jan 16 18:51 veth9ca63e79 -> ../../devices/virtual/net/veth9ca63e79
lrwxrwxrwx 1 root root 0 jan 16 18:51 vethc273b1aa -> ../../devices/virtual/net/vethc273b1aa
lrwxrwxrwx 1 root root 0 jan 16 18:51 vethdbd492f6 -> ../../devices/virtual/net/vethdbd492f6
root@router-server:/sys/class/net# incus exec openwrt -- ls -l /sys/class/net/
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-base -> ../../devices/virtual/net/br-base
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-base.10 -> ../../devices/virtual/net/br-base.10
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-base.11 -> ../../devices/virtual/net/br-base.11
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-base.12 -> ../../devices/virtual/net/br-base.12
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-base.13 -> ../../devices/virtual/net/br-base.13
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-base.14 -> ../../devices/virtual/net/br-base.14
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-base.15 -> ../../devices/virtual/net/br-base.15
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-dmz -> ../../devices/virtual/net/br-dmz
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-guest -> ../../devices/virtual/net/br-guest
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-iot -> ../../devices/virtual/net/br-iot
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-lan -> ../../devices/virtual/net/br-lan
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-services -> ../../devices/virtual/net/br-services
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 br-untrusted -> ../../devices/virtual/net/br-untrusted
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 eth0 -> ../../devices/pci0000:00/0000:00:1c.4/0000:02:00.0/net/eth0
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 eth1 -> ../../devices/pci0000:00/0000:00:1c.5/0000:03:00.0/net/eth1
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 eth2 -> ../../devices/pci0000:00/0000:00:1c.6/0000:04:00.0/net/eth2
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 eth3-wan -> ../../devices/pci0000:00/0000:00:1c.7/0000:05:00.0/net/eth3-wan
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 local-dmz -> ../../devices/virtual/net/local-dmz
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 local-iot -> ../../devices/virtual/net/local-iot
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 local-lan -> ../../devices/virtual/net/local-lan
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 local-services -> ../../devices/virtual/net/local-services
lrwxrwxrwx    1 root     root             0 Jan 16 18:51 local-untrusted -> ../../devices/virtual/net/local-untrusted

Again, it’s there. I’m quite curious now for who is renaming that interface and why would it happen in openwrt, debian and ubuntu and does it happen exactly when I restart the instance?

9

Simple, that check was added yesterday and isn’t in any current Incus stable builds :slight_smile:

If you were running a version of Incus with the code you’re looking at, you would have gotten the new error message I introduced while debugging a similar issue (OpenWRT with a Wifi card).

10

Yeah, debugging with @mcondarelli has showed that OpenWRT does actively rename interfaces both on startup and shutdown, so that’s definitely a bit of a problem.

Ubuntu or Debian usually do not do that, so those are a bit weirder.

Do you have the incusd.log errors you get when shutting down an Ubuntu or Debian container?
I wonder what device has gone missing in this case. Also the incus config show --expanded NAME for that instance just prior to the incus stop would be helpful.

Help setting up a bridge network adapter, separate physical nictype passthru
11

Is that discussion available in a public forum so that I can follow how you figured out this renaming is going on?

They show up on the incusd.log I posted previously. The container name for the openwrt container is openwrt while de debian and ubuntu containers were named d1 and u1. Let me copy and paste some of those lines from my previous post here.

root@router-server:/var/log/incus# ls
d1  incusd.log	incusd.log.1  liked-foal  openwrt  precious-rhino  u1
root@router-server:/var/log/incus# cat incusd.log.1
(...)
time="2024-01-16T18:47:07-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 709 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-16T18:47:08-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 709 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-16T18:47:08-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 709 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=d1 instanceType=container project=default
(...)
time="2024-01-16T18:48:57-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1257/fd/4 707 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=u1 instanceType=container project=default
time="2024-01-16T18:48:57-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1257/fd/4 707 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=u1 instanceType=container project=default
(...)

I will run the same experiments again and post the results here after I’m done. And just a small question, has the openwrt container image been changed recently in any way? I ask this because I can’t figure out how to reproduce the same error I’m getting on my router-server machine inside a VM.

12

We’ve been talking over the past few days with @mcondarelli on both IRC (#lxc) and on Github through incusd/forknet: Handle wifi detach by stgraber · Pull Request #395 · lxc/incus · GitHub and `incus stop <instance>` does not seem to release `nictype=physical` devices · Issue #385 · lxc/incus · GitHub.

The focus has been primarily on wifi devices but they exhibit the same kind of issue.

It’s definitely odd that Ubuntu is seen having the same problem here.
Were you using the exact same config as your openwrt container above?

As for changes to the openwrt container images. They are built and published daily, so if something changed in the openwrt repository, that would have gotten picked up. But we didn’t make any other change on our end, so no changes to what’s included in the image or its default configuration.

13

Before posting the results of the commands you mentioned, I’ll first post something I found odd which might be related.

Due to the current configuration in which I need to reboot to get the correct network interface names back, I was trying to find an alternative way to get the network interface names correct. At some point, I had enp2s0 renamed to physT5THFL (what I mentioned before that it did not happen to the first interface happened this time). I tried to rename it back manually by running sudo ip link set physT5THFL name enp2s0 and the command would return RTNETLINK answers: File exists. I could rename the interface to anything else, so renaming the interface was not an issue. It was particularly to the previous names those interfaces had which I could not rename them,

I have not read those posts entirely, but I saw the error was also happening on Debian 12 as a host server. It might be that OpenWRT is not renaming the interface, it could be something on the Debian side of the network configuration which I know is different between an iso installation and a standard images: image running on a VM.

14

When that File exists error happens, what do you get when you do ip link show dev enp2s0?

15

I’ll have to go back to the machine to test that. Before I do that, I have to post something I found while searching online just now.

I found an old post with the same type of configuration as mine. The same error occurs on an Ubuntu host by the way. There is an explanation of where the renaming issue is occurring and it’s actually something done by lxc.

A solution was provided on the host server configuration which was not confirmed to work by the original author, but it was confirmed by someone else afterwards

16

I stopped openwrt and only one interface had the renaming issue, which was enp5s0.

victoitor@router-server:~$ sudo ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1d brd ff:ff:ff:ff:ff:ff
4: enp4s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1e brd ff:ff:ff:ff:ff:ff
5: physAYo8YM: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
    altname enp5s0
6: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether e2:da:41:4f:38:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fd95:6f3:b94e:0:e0da:41ff:fe4f:38c4/64 scope global dynamic mngtmpaddr 
       valid_lft forever preferred_lft forever
    inet6 fe80::e0da:41ff:fe4f:38c4/64 scope link 
       valid_lft forever preferred_lft forever
7: iot: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:2b:1b:e1 brd ff:ff:ff:ff:ff:ff
8: services: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:3b:c5:a7 brd ff:ff:ff:ff:ff:ff
9: untrusted: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:09:29:e1 brd ff:ff:ff:ff:ff:ff
10: dmz: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:c9:b0:7a brd ff:ff:ff:ff:ff:ff

Same thing with trying to rename.

victoitor@router-server:~$ sudo ip link set physAYo8YM name enp5s0
RTNETLINK answers: File exists.

What you asked for:

victoitor@router-server:~$ sudo ip link show dev enp5s0
5: physAYo8YM: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
    altname enp5s0

So it does recognize the interface. Maybe use this command to get the new name of the interface in case this bad renaming happens?

17

Btw, you asked for these previously and I did not send them yet.

This is what I got from incus config show openwrt --expanded prior to stopping it.

architecture: x86_64
config:
  boot.autostart: "true"
  boot.autostart.priority: "10"
  image.architecture: amd64
  image.description: Openwrt 23.05 amd64 (20240115_11:57)
  image.os: Openwrt
  image.release: "23.05"
  image.serial: "20240115_11:57"
  image.type: squashfs
  image.variant: default
  volatile.base_image: f17085e23771357cdf5e06d99c418707dfdeb0c40a50a97602f2ad7d35ef6d86
  volatile.cloud-init.instance-id: 6463fddd-1c48-4ebd-a286-c1be00c19e1b
  volatile.eth0.host_name: enp2s0
  volatile.eth0.last_state.created: "false"
  volatile.eth0.last_state.hwaddr: 60:be:b4:02:59:1c
  volatile.eth0.last_state.mtu: "1500"
  volatile.eth1.host_name: enp3s0
  volatile.eth1.last_state.created: "false"
  volatile.eth1.last_state.hwaddr: 60:be:b4:02:59:1d
  volatile.eth1.last_state.mtu: "1500"
  volatile.eth2.host_name: enp4s0
  volatile.eth2.last_state.created: "false"
  volatile.eth2.last_state.hwaddr: 60:be:b4:02:59:1e
  volatile.eth2.last_state.mtu: "1500"
  volatile.eth3-wan.host_name: enp5s0
  volatile.eth3-wan.last_state.created: "false"
  volatile.eth3-wan.last_state.hwaddr: 60:be:b4:02:59:1f
  volatile.eth3-wan.last_state.mtu: "1500"
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.local-dmz.host_name: veth9ca63e79
  volatile.local-dmz.hwaddr: 00:16:3e:95:05:01
  volatile.local-iot.host_name: vethc273b1aa
  volatile.local-iot.hwaddr: 00:16:3e:71:5d:f4
  volatile.local-lan.host_name: vethdbd492f6
  volatile.local-lan.hwaddr: 00:16:3e:4b:a4:cb
  volatile.local-services.host_name: veth5c834edf
  volatile.local-services.hwaddr: 00:16:3e:39:54:b9
  volatile.local-untrusted.host_name: veth319c6919
  volatile.local-untrusted.hwaddr: 00:16:3e:f7:cb:2e
  volatile.uuid: a3099060-8bf3-4a6e-8e30-f9d841c3d478
  volatile.uuid.generation: a3099060-8bf3-4a6e-8e30-f9d841c3d478
devices:
  eth0:
    name: eth0
    nictype: physical
    parent: enp2s0
    type: nic
  eth1:
    name: eth1
    nictype: physical
    parent: enp3s0
    type: nic
  eth2:
    name: eth2
    nictype: physical
    parent: enp4s0
    type: nic
  eth3-wan:
    name: eth3-wan
    nictype: physical
    parent: enp5s0
    type: nic
  local-dmz:
    name: local-dmz
    network: dmz
    type: nic
  local-iot:
    name: local-iot
    network: iot
    type: nic
  local-lan:
    name: local-lan
    nictype: bridged
    parent: br0
    type: nic
  local-services:
    name: local-services
    network: services
    type: nic
  local-untrusted:
    name: local-untrusted
    network: untrusted
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- router
stateful: false
description: ""

And this is incus.log after running the stop command.

time="2024-01-18T08:44:31-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1185/fd/4 707 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default

This is what I got from incus config show d1 --expanded prior to stopping it.

architecture: x86_64
config:
  boot.autostart.priority: "10"
  image.architecture: amd64
  image.description: Debian bookworm amd64 (20240118_05:24)
  image.os: Debian
  image.release: bookworm
  image.serial: "20240118_05:24"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 7596cba005f34eafed284f5cced900a216114761f2dc05b8ee406016d2a40549
  volatile.cloud-init.instance-id: c054a3e6-f64d-4068-aa79-1e765c6093b0
  volatile.eth0.host_name: enp2s0
  volatile.eth0.last_state.created: "false"
  volatile.eth0.last_state.hwaddr: 60:be:b4:02:59:1c
  volatile.eth0.last_state.mtu: "1500"
  volatile.eth1.host_name: enp3s0
  volatile.eth1.last_state.created: "false"
  volatile.eth1.last_state.hwaddr: 60:be:b4:02:59:1d
  volatile.eth1.last_state.mtu: "1500"
  volatile.eth2.host_name: enp4s0
  volatile.eth2.last_state.created: "false"
  volatile.eth2.last_state.hwaddr: 60:be:b4:02:59:1e
  volatile.eth2.last_state.mtu: "1500"
  volatile.eth3-wan.host_name: enp5s0
  volatile.eth3-wan.last_state.created: "false"
  volatile.eth3-wan.last_state.hwaddr: 60:be:b4:02:59:1f
  volatile.eth3-wan.last_state.mtu: "1500"
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.local-dmz.host_name: vethf4ab808a
  volatile.local-dmz.hwaddr: 00:16:3e:f6:b1:33
  volatile.local-iot.host_name: veth109c7080
  volatile.local-iot.hwaddr: 00:16:3e:fd:ab:02
  volatile.local-lan.host_name: veth9fd340ee
  volatile.local-lan.hwaddr: 00:16:3e:53:ad:7a
  volatile.local-services.host_name: vethda28563f
  volatile.local-services.hwaddr: 00:16:3e:ea:17:54
  volatile.local-untrusted.host_name: veth14242bd1
  volatile.local-untrusted.hwaddr: 00:16:3e:95:5a:40
  volatile.uuid: 17922eb9-c7ee-4048-a54d-a06790a70f7b
  volatile.uuid.generation: 17922eb9-c7ee-4048-a54d-a06790a70f7b
devices:
  eth0:
    name: eth0
    nictype: physical
    parent: enp2s0
    type: nic
  eth1:
    name: eth1
    nictype: physical
    parent: enp3s0
    type: nic
  eth2:
    name: eth2
    nictype: physical
    parent: enp4s0
    type: nic
  eth3-wan:
    name: eth3-wan
    nictype: physical
    parent: enp5s0
    type: nic
  local-dmz:
    name: local-dmz
    network: dmz
    type: nic
  local-iot:
    name: local-iot
    network: iot
    type: nic
  local-lan:
    name: local-lan
    nictype: bridged
    parent: br0
    type: nic
  local-services:
    name: local-services
    network: services
    type: nic
  local-untrusted:
    name: local-untrusted
    network: untrusted
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- router
stateful: false
description: ""

I also ran incus exec d1 -- ip a prior to stopping the container.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 60:be:b4:02:59:1d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::62be:b4ff:fe02:591d/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1e brd ff:ff:ff:ff:ff:ff
5: eth3-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::62be:b4ff:fe02:591f/64 scope link 
       valid_lft forever preferred_lft forever
11: local-dmz@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:f6:b1:33 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fef6:b133/64 scope link 
       valid_lft forever preferred_lft forever
13: local-iot@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:fd:ab:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fefd:ab02/64 scope link 
       valid_lft forever preferred_lft forever
15: local-lan@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:53:ad:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fe53:ad7a/64 scope link 
       valid_lft forever preferred_lft forever
17: local-services@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:ea:17:54 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:feea:1754/64 scope link 
       valid_lft forever preferred_lft forever
19: local-untrusted@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:95:5a:40 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fe95:5a40/64 scope link 
       valid_lft forever preferred_lft forever

And this is incus.log after running the stop command.

time="2024-01-18T08:44:31-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1185/fd/4 707 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-18T08:59:23-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-18T08:59:23-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-18T08:59:44-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:43145->8.8.8.8:53: i/o timeout"
time="2024-01-18T08:59:44-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:43145->8.8.8.8:53: i/o timeout"
time="2024-01-18T09:03:22-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1277/fd/4 711 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-18T09:17:53-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-18T09:17:53-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-18T09:18:14-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:52447->8.8.8.8:53: i/o timeout"
time="2024-01-18T09:18:14-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:52447->8.8.8.8:53: i/o timeout"
time="2024-01-18T09:20:12-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 707 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-18T09:20:12-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 707 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-18T09:20:12-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 707 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-18T09:20:13-03:00" level=error msg="Failed to stop device" device=eth0 err="Failed to detach interface: \"eth0\" to \"enp2s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 707 eth0 enp2s0: exit status 1 (Error: Failed to run: ip addr flush dev eth0: exit status 1 (Device \"eth0\" does not exist.))" instance=d1 instanceType=container project=default

This is what I got from incus config show u1 --expanded prior to stopping it.

architecture: x86_64
config:
  boot.autostart.priority: "10"
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20240118_07:42)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20240118_07:42"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 6f64ec2a20c2ed2d531f04c131b6e3d2ceb76d2b5a215f2d2bf0a6b800f53cc1
  volatile.cloud-init.instance-id: 0dbdefa9-8037-4e55-b35c-e21fcf9d743a
  volatile.eth0.host_name: enp2s0
  volatile.eth0.last_state.created: "false"
  volatile.eth0.last_state.hwaddr: 60:be:b4:02:59:1c
  volatile.eth0.last_state.mtu: "1500"
  volatile.eth1.host_name: enp3s0
  volatile.eth1.last_state.created: "false"
  volatile.eth1.last_state.hwaddr: 60:be:b4:02:59:1d
  volatile.eth1.last_state.mtu: "1500"
  volatile.eth2.host_name: enp4s0
  volatile.eth2.last_state.created: "false"
  volatile.eth2.last_state.hwaddr: 60:be:b4:02:59:1e
  volatile.eth2.last_state.mtu: "1500"
  volatile.eth3-wan.host_name: enp5s0
  volatile.eth3-wan.last_state.created: "false"
  volatile.eth3-wan.last_state.hwaddr: 60:be:b4:02:59:1f
  volatile.eth3-wan.last_state.mtu: "1500"
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.local-dmz.host_name: veth6eb6bece
  volatile.local-dmz.hwaddr: 00:16:3e:8a:36:da
  volatile.local-iot.host_name: vethb335e667
  volatile.local-iot.hwaddr: 00:16:3e:b4:ba:c8
  volatile.local-lan.host_name: veth38b407d4
  volatile.local-lan.hwaddr: 00:16:3e:50:ff:e2
  volatile.local-services.host_name: vetha3107c60
  volatile.local-services.hwaddr: 00:16:3e:a8:81:48
  volatile.local-untrusted.host_name: veth8f8a379f
  volatile.local-untrusted.hwaddr: 00:16:3e:0d:01:93
  volatile.uuid: af6a99ed-12ad-4e8c-b824-6772d354acc2
  volatile.uuid.generation: af6a99ed-12ad-4e8c-b824-6772d354acc2
devices:
  eth0:
    name: eth0
    nictype: physical
    parent: enp2s0
    type: nic
  eth1:
    name: eth1
    nictype: physical
    parent: enp3s0
    type: nic
  eth2:
    name: eth2
    nictype: physical
    parent: enp4s0
    type: nic
  eth3-wan:
    name: eth3-wan
    nictype: physical
    parent: enp5s0
    type: nic
  local-dmz:
    name: local-dmz
    network: dmz
    type: nic
  local-iot:
    name: local-iot
    network: iot
    type: nic
  local-lan:
    name: local-lan
    nictype: bridged
    parent: br0
    type: nic
  local-services:
    name: local-services
    network: services
    type: nic
  local-untrusted:
    name: local-untrusted
    network: untrusted
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- router
stateful: false
description: ""

I also ran incus exec u1 -- ip a prior to stopping the container.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1c brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 60:be:b4:02:59:1d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::62be:b4ff:fe02:591d/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 60:be:b4:02:59:1e brd ff:ff:ff:ff:ff:ff
5: eth3-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::62be:b4ff:fe02:591f/64 scope link 
       valid_lft forever preferred_lft forever
11: local-dmz@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:8a:36:da brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fe8a:36da/64 scope link 
       valid_lft forever preferred_lft forever
13: local-iot@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:b4:ba:c8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:feb4:bac8/64 scope link 
       valid_lft forever preferred_lft forever
15: local-lan@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:50:ff:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fe50:ffe2/64 scope link 
       valid_lft forever preferred_lft forever
17: local-services@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:a8:81:48 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fea8:8148/64 scope link 
       valid_lft forever preferred_lft forever
19: local-untrusted@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:0d:01:93 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::216:3eff:fe0d:193/64 scope link 
       valid_lft forever preferred_lft forever

And this is incus.log after running the stop command.

time="2024-01-18T08:44:31-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1185/fd/4 707 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-18T08:59:23-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-18T08:59:23-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-18T08:59:44-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:43145->8.8.8.8:53: i/o timeout"
time="2024-01-18T08:59:44-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:43145->8.8.8.8:53: i/o timeout"
time="2024-01-18T09:03:22-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1277/fd/4 711 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=openwrt instanceType=container project=default
time="2024-01-18T09:17:53-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-18T09:17:53-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-18T09:18:14-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:52447->8.8.8.8:53: i/o timeout"
time="2024-01-18T09:18:14-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:52447->8.8.8.8:53: i/o timeout"
time="2024-01-18T09:20:12-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 707 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-18T09:20:12-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 707 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-18T09:20:12-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 707 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-18T09:20:13-03:00" level=error msg="Failed to stop device" device=eth0 err="Failed to detach interface: \"eth0\" to \"enp2s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1258/fd/4 707 eth0 enp2s0: exit status 1 (Error: Failed to run: ip addr flush dev eth0: exit status 1 (Device \"eth0\" does not exist.))" instance=d1 instanceType=container project=default
time="2024-01-18T09:30:47-03:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2024-01-18T09:30:47-03:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-01-18T09:31:09-03:00" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:53384->8.8.8.8:53: i/o timeout"
time="2024-01-18T09:31:09-03:00" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/.yaml\": lookup images.linuxcontainers.org on 8.8.8.8:53: read udp 10.0.0.10:53384->8.8.8.8:53: i/o timeout"
time="2024-01-18T09:32:50-03:00" level=error msg="Failed to stop device" device=eth3-wan err="Failed to detach interface: \"eth3-wan\" to \"enp5s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1248/fd/4 705 eth3-wan enp5s0: exit status 1 (Error: Failed to run: ip addr flush dev eth3-wan: exit status 1 (Device \"eth3-wan\" does not exist.))" instance=u1 instanceType=container project=default
time="2024-01-18T09:32:50-03:00" level=error msg="Failed to stop device" device=eth2 err="Failed to detach interface: \"eth2\" to \"enp4s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1248/fd/4 705 eth2 enp4s0: exit status 1 (Error: Failed to run: ip addr flush dev eth2: exit status 1 (Device \"eth2\" does not exist.))" instance=u1 instanceType=container project=default
time="2024-01-18T09:32:50-03:00" level=error msg="Failed to stop device" device=eth1 err="Failed to detach interface: \"eth1\" to \"enp3s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1248/fd/4 705 eth1 enp3s0: exit status 1 (Error: Failed to run: ip addr flush dev eth1: exit status 1 (Device \"eth1\" does not exist.))" instance=u1 instanceType=container project=default
time="2024-01-18T09:32:50-03:00" level=error msg="Failed to stop device" device=eth0 err="Failed to detach interface: \"eth0\" to \"enp2s0\": Failed to run: /opt/incus/bin/incusd forknet detach -- /proc/1248/fd/4 705 eth0 enp2s0: exit status 1 (Error: Failed to run: ip addr flush dev eth0: exit status 1 (Device \"eth0\" does not exist.))" instance=u1 instanceType=container project=default
18

I’m not sure you have checked the old message I linked to here. In my view, I think it was somewhat well explained that the renaming was done by lxc and not by the container itself. It seems a host config is not allowing for the renaming of the container.

A simple fix at least to allow restarting the container would be to have the ip command get the new name of the interface in case there was a renaming error like how I got it from here:

victoitor@router-server:~$ sudo ip link show dev enp5s0
5: physAYo8YM: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 60:be:b4:02:59:1f brd ff:ff:ff:ff:ff:ff
    altname enp5s0

Something else which would be odd for openwrt to be renaming the interface is that it’s expected to run on systems which are turned off without a shutdown command. So it’s odd to rename something on shutdown itself.

Do you have a similar understanding to mine on what’s happening? Does it make sense to be the lxc renaming from the other post?

19

I’m seeing the same kind of problem with a VyOS guest VM on an Arch Linux host and nictype: physical passthrough for a quad-port NIC. I tried removing the keep argument from /usr/lib/systemd/network/99-default.link, but that did not resolve the issue with starting the VM after having shut it down. Is there anything else to try?

20

I wasn’t able to solve the problem, actually. So I don’t know…