Incus container unable to reach outside world

Incus
1

I’m having issues with reaching the outside world from my containers (ping google.com, Git, etc).

I’ve done hours of searching and reading various other forum posts, and tried the various suggestions, but have not been able to fix the issue.
I’d be very grateful for some assistance in getting this working.

Below are some of the debug commands that seem to be useful from the other similar posts!
Please let me know of any other details that may be required.

Many thanks!

Hetzner Cloud.

Host and container both Ubuntu 24.04, container via:

incus launch images:ubuntu/24.04 rpc

Incus version:

testhost@testhost-one:~$ incus --version
6.3

Host

Firewall / rules status:

testhost@testhost-one:~$ sudo ufw status
Status: inactive


testhost@testhost-one:~$ sudo iptables -t nat -L -n -v --line-numbers
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination


testhost@testhost-one:~$ ps aux | grep dnsmasq
incus      19195  0.0  0.1  14472  5248 ?        Ss   Aug04   0:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=incusbr0 --dhcp-rapid-commit --no-negcache --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.185.39.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/lib/incus/networks/incusbr0/dnsmasq.leases --dhcp-hostsfile=/var/lib/incus/networks/incusbr0/dnsmasq.hosts --dhcp-range 10.185.39.2,10.185.39.254,1h --listen-address=fd42:5ea3:6855:6e02::1 --enable-ra --dhcp-range ::,constructor:incusbr0,ra-stateless,ra-names -s incus --interface-name _gateway.incus,incusbr0 -S /incus/ --conf-file=/var/lib/incus/networks/incusbr0/dnsmasq.raw -u incus -g incus
wonders+   21621  0.0  0.0   6544  2304 pts/0    S+   12:59   0:00 grep --color=auto dnsmasq


testhost@testhost-one:~$ ip -4 route show
10.0.0.0/16 via 10.0.0.1 dev enp7s0 proto dhcp src 10.0.0.6 metric 1003 mtu 1450
10.0.0.1 dev enp7s0 proto dhcp scope link src 10.0.0.6 metric 1003 mtu 1450
10.185.39.0/24 dev incusbr0 proto kernel scope link src 10.185.39.1
169.254.169.254 via 172.31.1.1 dev eth0 proto dhcp src 100.65.193.187 metric 100
172.31.1.1 dev eth0 proto dhcp scope link src 100.65.193.187 metric 100

Pinging google.com (working):

testhost@testhost-one:~$ ping google.com
PING google.com (2a00:1450:4001:812::200e) 56 data bytes
64 bytes from fra16s52-in-x0e.1e100.net (2a00:1450:4001:812::200e): icmp_seq=1 ttl=115 time=6.38 ms
64 bytes from fra16s52-in-x0e.1e100.net (2a00:1450:4001:812::200e): icmp_seq=2 ttl=115 time=6.77 ms

Incus info:

testhost@testhost-one:~$ incus info --show-log rpc
Name: rpc
Status: RUNNING
Type: container
Architecture: x86_64
PID: 18217
Created: 2024/08/04 23:16 UTC
Last Used: 2024/08/04 23:16 UTC
Started: 2024/08/04 23:16 UTC

Resources:
  Processes: 12
  CPU usage:
    CPU usage (in seconds): 19
  Memory usage:
    Memory (current): 438.85MiB
  Network usage:
    eth0:
      Type: broadcast
      State: UP
      Host interface: veth5f2c0c8a
      MAC address: 00:16:3e:2a:d6:0d
      MTU: 1500
      Bytes received: 15.92MB
      Bytes sent: 742.17kB
      Packets received: 8568
      Packets sent: 6736
      IP addresses:
        inet:  10.185.39.39/24 (global)
        inet6: fd42:5ea3:6855:6e02:216:3eff:fe2a:d60d/64 (global)
        inet6: fe80::216:3eff:fe2a:d60d/64 (link)
    lo:
      Type: loopback
      State: UP
      MTU: 65536
      Bytes received: 10.69kB
      Bytes sent: 10.69kB
      Packets received: 84
      Packets sent: 84
      IP addresses:
        inet:  127.0.0.1/8 (local)
        inet6: ::1/128 (local)

Log:

Incus list:

+-------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME  |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+-------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| first | RUNNING | 10.185.39.190 (eth0) | fd42:5ea3:6855:6e02:216:3eff:fe0c:a71f (eth0) | CONTAINER | 0         |
+-------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| rpc   | RUNNING | 10.185.39.39 (eth0)  | fd42:5ea3:6855:6e02:216:3eff:fe2a:d60d (eth0) | CONTAINER | 0         |
+-------+---------+----------------------+-----------------------------------------------+-----------+-----------+

Incus network list:

testhost@testhost-one:~$ incus network list
+----------+----------+---------+----------------+---------------------------+-------------+---------+---------+
|   NAME   |   TYPE   | MANAGED |      IPV4      |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+----------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| enp7s0   | physical | false   |                |                           |             | 0       |         |
+----------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| eth0     | physical | false   |                |                           |             | 0       |         |
+----------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| incusbr0 | bridge   | true    | 10.185.39.1/24 | fd42:5ea3:6855:6e02::1/64 |             | 3       | CREATED |
+----------+----------+---------+----------------+---------------------------+-------------+---------+---------+
| lo       | loopback | false   |                |                           |             | 0       |         |
+----------+----------+---------+----------------+---------------------------+-------------+---------+---------+

resolvectl:

testhost@testhost-one:~$ resolvectl
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 2a01:4ff:ff00::add:1
       DNS Servers: 2a01:4ff:ff00::add:1 2a01:4ff:ff00::add:2

Link 3 (enp7s0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (incusbr0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (vethc20fe739)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 8 (veth5f2c0c8a)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Container

Pinging sibling container (working):

root@rpc:~# ping 10.0.0.5
PING 10.0.0.5 (10.0.0.5) 56(84) bytes of data.
64 bytes from 10.0.0.5: icmp_seq=1 ttl=62 time=5.91 ms
64 bytes from 10.0.0.5: icmp_seq=2 ttl=62 time=0.568 ms

From the container - pinging google.com (not working):

root@rpc:~# ping google.com
PING google.com (142.250.181.238) 56(84) bytes of data.
From _gateway.incus (10.185.39.1) icmp_seq=1 Destination Net Unreachable
From _gateway.incus (10.185.39.1) icmp_seq=2 Destination Net Unreachable

Other details from the container:

root@rpc:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:2a:d6:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.185.39.39/24 metric 100 brd 10.185.39.255 scope global dynamic eth0
       valid_lft 2490sec preferred_lft 2490sec
    inet6 fd42:5ea3:6855:6e02:216:3eff:fe2a:d60d/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe2a:d60d/64 scope link
       valid_lft forever preferred_lft forever

root@rpc:~# ip r
default via 10.185.39.1 dev eth0 proto dhcp src 10.185.39.39 metric 100
10.185.39.0/24 dev eth0 proto kernel scope link src 10.185.39.39 metric 100
10.185.39.1 dev eth0 proto dhcp scope link src 10.185.39.39 metric 100

resolvectl:

root@rpc:~# resolvectl
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 7 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.185.39.1
       DNS Servers: 10.185.39.1 fd42:5ea3:6855:6e02::1 fe80::216:3eff:fea9:18cf
        DNS Domain: incus
2

nft list ruleset may help here?
Also incus network show incusbr0

3

Thanks for your reply @stgraber, here’s the following additional details:

nft list ruleset:

testhost@testhost-one:~$ sudo nft list ruleset
table inet incus {
        chain pstrt.incusbr0 {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr 10.185.39.0/24 ip daddr != 10.185.39.0/24 masquerade
                ip6 saddr fd42:5ea3:6855:6e02::/64 ip6 daddr != fd42:5ea3:6855:6e02::/64 masquerade
        }
}
table ip filter {
        chain INPUT {
                type filter hook input priority filter; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority filter; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
        }
}
table ip6 filter {
        chain INPUT {
                type filter hook input priority filter; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority filter; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
        }
}

incus network show incusbr0:

testhost@testhost-one:~$ incus network show incusbr0
config:
  ipv4.address: 10.185.39.1/24
  ipv4.firewall: "false"
  ipv4.nat: "true"
  ipv6.address: fd42:5ea3:6855:6e02::1/64
  ipv6.firewall: "false"
  ipv6.nat: "true"
description: ""
name: incusbr0
type: bridge
used_by:
- /1.0/instances/first
- /1.0/instances/rpc
- /1.0/profiles/default
managed: true
status: Created
locations:
- none
project: default
4

Can you show ip -4 r get 142.250.181.238 on the host system?

5 
testhost@testhost-one:~$ ip -4 r get 142.250.181.238
RTNETLINK answers: Network is unreachable
6

Okay, so your host system doesn’t seem to have IPv4 connectivity which is why your containers also don’t.

7

Thanks for clarifying.
Is it possible for connectivity to work IPv6-only, like it does on the host?

For example, ping google.com works on the host even though it is IPv6-only, and apt install works in the container at the moment.

8

Yes, but for this to work properly you need to do one of two things:

  • Switch to a non ULA subnet for your instances
  • Stop providing IPv4 to your instances

With normal global subnets, Linux will prefer IPv6 over IPv4, but when on a ULA subnet, it will prefer IPv4 over IPv6. In your case it’s basically seeing that ULA subnet and sees what looks like working IPv4 (there’s a default route) and so goes for an IPv4 connection.

9
  • Switch to a non ULA subnet for your instances

Thanks, are you able to provide instructions on how to do this?

10

incus network set incusbr0 ipv6.nat=2001:db8:1234:1234::1/64

(2001:db8 is the prefix reserved for examples/documentation, so it’s not a ULA and is also not stealing someone else’s address space)

11

For any future readers, the command is:

incus network set incusbr0 ipv6.address=2001:db8:1234:1234::1/64

Many thanks for your help and time Stéphane!