Launching GROK3 Error: Failed instance creation: Failed pre-start check for device "eth0": Network "incusbr0" unavailable on this server

Incus
1

incus launch images:ubuntu/23.04 GROK3 --target RRAI
Launching GROK3
Error: Failed instance creation: Failed pre-start check for device “eth0”: Network “incusbr0” unavailable on this server

root@RRAI:/home/RR2020# @ network list
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
|   NAME   |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+       
| 10.0.0.0 | bridge   | YES     | 10.119.230.1/24 | fd42:266e:d72f:8869::1/64 |             | 0       | CREATED |       
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+       
| eno2     | physical | NO      |                 |                           |             | 0       |         |       
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+       
| incusbr0 | bridge   | YES     | 10.126.149.1/24 | fd42:ecf:4f02:b679::1/64  |             | 34      | CREATED |       
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
2

Do you see incusbr0 in ip link show on that machine?
Any error message related to that network in /var/log/incus/incusd.log?

3

ip link show | grep incusbr0 SHOW NOTHING
root@RRAI:/home/RR2020# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 9c:2d:cd:11:30:25 brd ff:ff:ff:ff:ff:ff
altname enp0s31f6

"time=“2024-03-22T11:42:01-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 2 ("dnsmasq: failed to create listening socket for 10.126.149.1: Address already in use")” ntime=“2024-03-22T11:43:02-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 2 ("dnsmasq: failed to create listening socket for 10.119.230.1: Address already in use")” ntime=“2024-03-22T11:43:02-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 2 ("dnsmasq: failed to create listening socket for 10.126.149.1: Address already in use")” ntime=“2024-03-22T11:44:02-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 2 ("dnsmasq: failed to create listening socket for 10.119.230.1: Address already in use")” ntime=“2024-03-22T11:44:03-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 2 ("dnsmasq: failed to create listening socket for 10.126.149.1: Address already in use")” n
"
dnsmasq: failed to create listening socket for 10.119.230.1: Address already in use

" @ cluster list
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| Q5 | https://192.168.1.27:8443 | database | x86_64 | default | | ONLINE | Fully operational |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| RRAI | https://192.168.1.50:8443 | database | x86_64 | default | | ONLINE | Fully operational |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| RRDB2 | https://192.168.1.26:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| RRSOC | https://192.168.1.18:8443 | database-standby | x86_64 | default | | ONLINE | Fully operational |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| RRTEAMS | https://192.168.1.25:8443 | database-standby | x86_64 | default | | ONLINE | Fully operational |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| mail.rrmails.com | https://192.168.1.24:8443 | | x86_64 | default | | ONLINE | Fully operational |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| rrcloud | https://192.168.1.22:8443 | | x86_64 | default | | ONLINE | Fully operational |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| rrterabk | https://192.168.1.15:8443 | | x86_64 | default | | ONLINE | Fully operational |
±-----------------±--------------------------±-----------------±-------------±---------------±------------±-------±------------------+ "

Cluster member seems to be fine, others are working ok, something related to this one only.

4

Can you show netstat -lnp on the system?
It sounds like you have something already listening on port 53 or 67 on that IP.

5

‘’’ ‘’'netstat -lnp | grep 53
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 192.168.1.50:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:53 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
tcp6 0 0 ::1:953 :::* LISTEN 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 192.168.1.50:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 2664023/named
udp 0 0 0.0.0.0:5353 0.0.0.0:* 928/avahi-daemon: r
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named
udp6 0 0 ::1:53 :::* 2664023/named

6

Nothing on 67

7

Right, so you have bind9/named installed on the system and listening on every interface, that will prevent Incus from starting dnsmasq.

Either remove bind9/named from the system or configure it to only listen on the interfaces it actually is needed on.

8

Thank you

9

I removed bind9 and rebooted

netstat -lnp | grep 53
udp 0 0 0.0.0.0:5353 0.0.0.0:* 928/avahi-daemon: r
udp6 0 0 :::5353 :::* 928/avahi-daemon: r
unix 2 [ ACC ] STREAM LISTENING 40530 1836/systemd /run/user/128/gnupg/S.gpg-agent.ssh
unix 2 [ ACC ] STREAM LISTENING 40532 1836/systemd /run/user/128/gnupg/S.gpg-agent
unix 2 [ ACC ] STREAM LISTENING 40534 1836/systemd /run/user/128/pipewire-0
unix 2 [ ACC ] STREAM LISTENING 40536 1836/systemd /run/user/128/pk-debconf-socket
unix 2 [ ACC ] STREAM LISTENING 40538 1836/systemd /run/user/128/pulse/native

@ start GROK3
Error: Failed pre-start check for device “eth0”: Network “incusbr0” unavailable on this server
Try incus info --show-log GROK3 for more info

Unfortunately no go/

from Logs

“2024-03-22T12:37:30-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 5 ("dnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll")” network=itime=“2024-03-22T12:37:37-04:00” level=warning msg=“Rejecting heartbeat request as shutting down”
time=“2024-03-22T12:37:37-04:00” level=warning msg=“Rejecting heartbeat request as shutting down”
time=“2024-03-22T12:37:37-04:00” level=error msg=“Failed to get storage pool” err=“Required tool ‘zpool’ is missing” pool=Q5Pool
time=“2024-03-22T12:38:10-04:00” level=warning msg=“Dqlite: attempt 1: server 192.168.1.15:8443: dial: Failed connecting to HTTP endpoint "192.168.1.15:8443": dial tcp 192.168.1.15:8443: connect: no route to host”
time=“2024-03-22T12:38:13-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 5 ("dnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll")” network=1time=“2024-03-22T12:38:13-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 5 ("dnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll")” network=itime=“2024-03-22T12:38:13-04:00” level=warning msg=“Failed to update instance types: Get "https://images.linuxcontainers.org/meta/instance-types/.yaml\”: lookup images.linuxcontainers.org on [::1]:53: read udp [::1]:54248->[::1]:53: read: connection rtime=“2024-03-22T12:38:13-04:00” level=error msg=“Failed updating instance types” err=“Get "https://images.linuxcontainers.org/meta/instance-types/.yaml\”: lookup images.linuxcontainers.org on [::1]:53: read udp [::1]:54248->[::1]:53: read: connectiontime=“2024-03-22T12:39:13-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 5 ("dnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll")” network=1time=“2024-03-22T12:39:13-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 5 ("dnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll")” network=itime=“2024-03-22T12:40:14-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 5 ("dnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll")” network=1time=“2024-03-22T12:40:14-04:00” level=error msg=“Failed initializing network” err=“Failed starting: The DNS and DHCP service exited prematurely: exit status 5 ("dnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll")” network=i

10

The gold bits out of the logs are:

  1. dnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll.
  2. Required tool ‘zpool’ is missing

These mean:

  1. This file was removed and messed up with the system. DNS should not work system-wide.
  2. zfsutils-linux not installed.