Issue adding nic as bridge to parent externally defined open vswitch bridge

yajrendrag · September 20, 2024, 5:47pm

I am trying out incus. i installed it on ubuntu 22.04 where i already have lxd installed too. When i did the admin init, it asked if i wanted to configure a new bridge to which i answered no and then asked if i wanted to use an existing bridge, which i answered yes and gave it my bridge name of br0.

Now this is the same bridge that lxd uses as it’s default, where it works fine.

but if i create a new container it doesn’t get an ip address from my network - whereas in lxd the same process results in the container getting an ip assigned from dhcp on my network. EDIT - this part is fixed - getting an IP with eth0 defined from the default profile works now - somehow my bridge had lost it’s IP address.

if i override the eth0 nic by issuing a new add command (which i guess incus does instead of doing an overt remove and readd) - e.g., incus config device add test-container eth0 nic nictype=bridged parent=br0 and then restart the container i get this error:

Error: Failed to start device "eth0": Failed to connect to OVS: failed to connect to unix:///run/openvswitch/db.sock: listdbs failure - unexpected EOF

The default profile for incus shows the exact same default eth0 device config as lxd (which has nictype of macvlan, but ultimately i want bridged) - and doing the above steps in lxd works.

the only other difference i can see is that when i do incus network list it doesn’t show the br0 network, whereas lxc network list does show br0 (as a nonmanaged network).

I have tried other solutions like trying to create an an uplink network to br0, but that doesn’t work either. I am wondering if br0 missing from the list of networks is the reason behind this not working as i expect or if there’s just a difference i haven’t figured out yet in incus networking and if so, what i need to get this to work. thanks for any help.

stgraber · September 21, 2024, 6:31pm

Can you show ip link show on the host?

yajrendrag · September 21, 2024, 10:09pm

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether b4:2e:99:a0:cd:2d brd ff:ff:ff:ff:ff:ff
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether 68:05:ca:1a:8b:59 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 4e:83:f5:3f:29:0d brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 68:05:ca:1a:8b:59 brd ff:ff:ff:ff:ff:ff
6: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 500
link/none
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:77:cf:e9 brd ff:ff:ff:ff:ff:ff
8: br-6e6ea1f41a98: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:d8:20:7c:d6 brd ff:ff:ff:ff:ff:ff
9: br-6f038191dbe7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:53:63:47:78 brd ff:ff:ff:ff:ff:ff
10: br-987898070adc: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:9c:56:38:af brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:b1:df:03:97 brd ff:ff:ff:ff:ff:ff
12: br-ee321761e1a7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:a8:8b:9f:27 brd ff:ff:ff:ff:ff:ff
13: br-51deebe8fcd6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:06:e0:f5:b8 brd ff:ff:ff:ff:ff:ff
14: br-5e56ead9fc0d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:97:36:64:d0 brd ff:ff:ff:ff:ff:ff
15: br-db5b7a1d254e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:eb:f1:72:f3 brd ff:ff:ff:ff:ff:ff
16: br-f2d2d3ff8e2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:1e:24:9f:40 brd ff:ff:ff:ff:ff:ff
18: veth821aeb5@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
link/ether ae:e7:98:fd:6f:98 brd ff:ff:ff:ff:ff:ff link-netnsid 0
20: veth9a1a884@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-db5b7a1d254e state UP mode DEFAULT group default
link/ether ce:c3:68:0c:a4:80 brd ff:ff:ff:ff:ff:ff link-netnsid 15
22: vethf36bdec@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-51deebe8fcd6 state UP mode DEFAULT group default
link/ether fa:b1:38:68:b3:7d brd ff:ff:ff:ff:ff:ff link-netnsid 1
24: veth737d126@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether fe:ac:8b:90:c5:97 brd ff:ff:ff:ff:ff:ff link-netnsid 7
26: veth785ce99@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f2d2d3ff8e2e state UP mode DEFAULT group default
link/ether 66:2b:42:eb:d6:bd brd ff:ff:ff:ff:ff:ff link-netnsid 2
28: veth855dc6c@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ee321761e1a7 state UP mode DEFAULT group default
link/ether ae:1c:63:86:f5:0e brd ff:ff:ff:ff:ff:ff link-netnsid 3
30: veth1915a37@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether 9e:66:a6:cc:35:5a brd ff:ff:ff:ff:ff:ff link-netnsid 8
32: mac3b62eb7e@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e2:26:0f:e4:b0:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 6
34: veth1c8f3f3@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
link/ether 8e:1a:a2:e1:70:2d brd ff:ff:ff:ff:ff:ff link-netnsid 5
36: vetha80d16a@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-db5b7a1d254e state UP mode DEFAULT group default
link/ether a2:0d:26:dc:ea:70 brd ff:ff:ff:ff:ff:ff link-netnsid 18
38: veth315d8e9@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether ca:2c:8b:67:6b:64 brd ff:ff:ff:ff:ff:ff link-netnsid 9
40: veth42e4c6c@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
link/ether ba:14:92:d0:a7:5d brd ff:ff:ff:ff:ff:ff link-netnsid 4
42: veth7096bc4@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-db5b7a1d254e state UP mode DEFAULT group default
link/ether 02:f3:ee:51:d3:da brd ff:ff:ff:ff:ff:ff link-netnsid 7
44: veth3bcba83@if43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether fa:ca:3c:54:d2:68 brd ff:ff:ff:ff:ff:ff link-netnsid 11
46: veth3fb3eee@if45: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-db5b7a1d254e state UP mode DEFAULT group default
link/ether ea:46:db:04:f7:8f brd ff:ff:ff:ff:ff:ff link-netnsid 8
48: veth87247f4@if47: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether b6:41:a5:36:a7:53 brd ff:ff:ff:ff:ff:ff link-netnsid 10
50: vethc71b895@if49: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-db5b7a1d254e state UP mode DEFAULT group default
link/ether e2:ac:ea:e7:58:26 brd ff:ff:ff:ff:ff:ff link-netnsid 9
52: veth4893b7c@if51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-db5b7a1d254e state UP mode DEFAULT group default
link/ether 3a:5a:16:ef:78:b5 brd ff:ff:ff:ff:ff:ff link-netnsid 11
54: vethf295513@if53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether 22:0e:b7:e9:4c:63 brd ff:ff:ff:ff:ff:ff link-netnsid 12
56: veth640fb44@if55: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether 1e:7a:a5:b2:e6:35 brd ff:ff:ff:ff:ff:ff link-netnsid 13
58: veth53fbf62@if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether 7a:f7:06:4d:bf:87 brd ff:ff:ff:ff:ff:ff link-netnsid 16
60: veth02e3fb1@if59: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether 32:8c:bd:0b:bb:77 brd ff:ff:ff:ff:ff:ff link-netnsid 17
62: mac4c46b770@if61: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether 82:ab:ed:38:ea:5a brd ff:ff:ff:ff:ff:ff link-netnsid 14
64: vethb23b251@if63: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-db5b7a1d254e state UP mode DEFAULT group default
link/ether e6:36:13:a5:af:1f brd ff:ff:ff:ff:ff:ff link-netnsid 16
66: veth6cbff97@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether 62:a1:3a:35:9b:17 brd ff:ff:ff:ff:ff:ff link-netnsid 15
68: vethccfd9dc@if67: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-db5b7a1d254e state UP mode DEFAULT group default
link/ether 26:3e:7d:7b:24:b9 brd ff:ff:ff:ff:ff:ff link-netnsid 17
70: veth2633705@if69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5e56ead9fc0d state UP mode DEFAULT group default
link/ether 22:2c:81:25:b6:cd brd ff:ff:ff:ff:ff:ff link-netnsid 18
72: mac8fd7e218@if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether 56:a1:be:8e:b2:fa brd ff:ff:ff:ff:ff:ff link-netnsid 19
90: incusbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 00:16:3e:70:7d:ce brd ff:ff:ff:ff:ff:ff

yajrendrag · September 21, 2024, 10:11pm

also, i know there are some conflicts with docker, but i have net.ipv4.ip_forward=1 in /etc/sysctl.conf & the incusbr0 i set up subsequently wondering if i’d be able to get that to uplink to br0…

also, a correction - host is ubuntu 24.04, not 22.04.

yajrendrag · September 24, 2024, 5:58pm

I installed a new ubuntu 24.04 system on other hardware, setup openvswitch the same way i have on the other hosts, installed incus and get the same results. So, it must be something i’m doing or not doing causing this result:

> incus config device add t eth0 nic nictype=bridged parent=br0
Error: Failed to start device "eth0": Failed to connect to OVS: failed to connect to unix:///run/openvswitch/db.sock: listdbs failure - unexpected EOF

it works fine with the default profile which defines eth0 as a macvlan type of nic.

ip link show results on this host:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 0c:4d:e9:af:b0:30 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether a8:86:dd:b5:5e:ed brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b6:4e:86:79:96:18 brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 0c:4d:e9:af:b0:30 brd ff:ff:ff:ff:ff:ff
6: br-4209bc6b147f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:88:58:0f:31 brd ff:ff:ff:ff:ff:ff
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:3a:51:8f:70 brd ff:ff:ff:ff:ff:ff
9: veth0b165df@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4209bc6b147f state UP mode DEFAULT group default 
    link/ether 96:7f:99:03:bd:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0

/etc/sysctl.conf has net.ipv4.ip_forward=1

@stgraber - any other ideas?
thanks.

—EDIT -more config details:

this config works with macvlan nictype - i get a dhcp address which i presume is from my hdcp server as the address is from that same pool (& it’s in the router’s network map - i just can’t reach it from the host and that is why i want a bridged type nic):

incus config show t --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (20240924_07:42)
  image.os: Ubuntu
  image.release: noble
  image.requirements.cgroup: v2
  image.serial: "20240924_07:42"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 50a6aacf01cd9b7d059b16af680ae17162b864028f5434f9a79559ddae2ec452
  volatile.cloud-init.instance-id: da6266c9-d9d5-458e-9149-28dcab73794f
  volatile.eth0.host_name: macdbb3cd81
  volatile.eth0.hwaddr: 00:16:3e:c8:03:4f
  volatile.eth0.last_state.created: "false"
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: e58269b3-e295-4b4d-a7fa-d3677404ff1e
  volatile.uuid.generation: e58269b3-e295-4b4d-a7fa-d3677404ff1e
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: br0
    type: nic
  root:
    path: /
    pool: loop
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

openvswitch detail:

jay@jay-macmini:~$ sudo ovs-vsctl show
3ac2b164-a1f5-42fc-814a-531012d03885
    Bridge br0
        Port enp1s0f0
            Interface enp1s0f0
        Port br0
            Interface br0
                type: internal
    ovs_version: "3.3.0"

but after stopping the container and i do: incus config device add t eth0 nic nictype=bridged parent=br0, then i get this upon restarting it:

Error: Failed to start device "eth0": Failed to connect to OVS: failed to connect to unix:///run/openvswitch/db.sock: listdbs failure - unexpected EOF

and here’s that resulting config:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (20240924_07:42)
  image.os: Ubuntu
  image.release: noble
  image.requirements.cgroup: v2
  image.serial: "20240924_07:42"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 50a6aacf01cd9b7d059b16af680ae17162b864028f5434f9a79559ddae2ec452
  volatile.cloud-init.instance-id: c3516ea7-8e4f-4cc7-9b3b-55951ffb3393
  volatile.eth0.hwaddr: 00:16:3e:5c:59:fe
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: e58269b3-e295-4b4d-a7fa-d3677404ff1e
  volatile.uuid.generation: e58269b3-e295-4b4d-a7fa-d3677404ff1e
devices:
  eth0:
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: loop
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

datiscum · September 24, 2024, 9:01pm

I had the same error message with Ubuntu 24.04.
But I installed incus from the Ubuntu repository.
The error was in the logfile /var/log/openvswitch/ovsdb-server.log:
2024-09-22T16:07:31.461Z|00017|jsonrpc|WARN|unix#13: received bad JSON-RPC message: message has unexpected member “jsonrpc”
2024-09-22T16:07:31.461Z|00018|reconnect|WARN|unix#13: connection dropped (Protocol error)

After I switched to the Zabbly repository the error disappeared.

yajrendrag · September 24, 2024, 9:09pm

thank you for the reply - i also installed from ubuntu 24.04 repo… will try zabbly…

andyl · September 24, 2024, 9:12pm

I also have good result installing Incus from Zabbly.

Maybe this will be useful - here is the netplan yaml file I used to configure the bridge on my Incus host running Ubuntu 22.04 server:

network: 
  version: 2
  renderer: networkd 
  ethernets: 
    enp114s0: 
      dhcp4: no
      dhcp6: no
  bridges: 
    bridge0: 
      interfaces: [enp114s0] 
      dhcp4: yes 
      openvswitch: {}

yajrendrag · September 24, 2024, 9:37pm

Yep - zabbly repo version of incus eliminated the issue. thanks all!