Incus on NixOS with tailscale on the side - no internet access for the containers

Hi there!

I have installed Incus on my RPI4 running NixOS, here is the config file part that is relevant:

  # Enable the OpenSSH daemon.
  services.openssh.enable = true;

  # Tailscale
  services.tailscale.enable = true;

  # Enable Incus
  virtualisation.incus.enable = true;

Apart from the hostname, I’m not defining any networking option.

I have initialized Incus using the default options, here is some network information:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:e7:5d:1d brd ff:ff:ff:ff:ff:ff
    inet 192.168.42.68/24 brd 192.168.42.255 scope global dynamic noprefixroute end0
       valid_lft 3077sec preferred_lft 2627sec
    inet6 fdb4:5d71:12a4:3942:afa:6ed4:fbbd:84d3/64 scope global temporary dynamic
       valid_lft 1627sec preferred_lft 1627sec
    inet6 fdb4:5d71:12a4:3942:c4e7:ea38:e6e5:cd11/64 scope global temporary deprecated dynamic
       valid_lft 1627sec preferred_lft 0sec
    inet6 fdb4:5d71:12a4:3942:dea6:32ff:fee7:5d1d/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 1627sec preferred_lft 1627sec
    inet6 fe80::dea6:32ff:fee7:5d1d/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether dc:a6:32:e7:5d:1e brd ff:ff:ff:ff:ff:ff
4: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 100.69.150.7/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0::4745:9607/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::5cef:a878:cae:c736/64 scope link stable-privacy proto kernel_ll
       valid_lft forever preferred_lft forever
5: incusbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:f2:e7:85 brd ff:ff:ff:ff:ff:ff
    inet 10.223.246.1/24 scope global incusbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:9dc9:1068:2f88::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fef2:e785/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
7: veth81f9586d@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP group default qlen 1000
    link/ether ca:80:d9:b7:10:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.167.150/16 brd 169.254.255.255 scope global noprefixroute veth81f9586d
       valid_lft forever preferred_lft forever
incus network list
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
|   NAME   |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| end0     | physical | NO      |                 |                           |             | 0       |         |
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| incusbr0 | bridge   | YES     | 10.223.246.1/24 | fd42:9dc9:1068:2f88::1/64 |             | 2       | CREATED |
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
| wlan0    | physical | NO      |                 |                           |             | 0       |         |
+----------+----------+---------+-----------------+---------------------------+-------------+---------+---------+
incus network show incusbr0
config:
  ipv4.address: 10.223.246.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:9dc9:1068:2f88::1/64
  ipv6.nat: "true"
description: ""
name: incusbr0
type: bridge
used_by:
- /1.0/instances/test
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination    
ts-input   all  --  anywhere             anywhere       
nixos-fw   all  --  anywhere             anywhere       

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination    
ts-forward  all  --  anywhere             anywhere      

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination    

Chain nixos-fw (1 references)
target     prot opt source               destination    
nixos-fw-accept  all  --  anywhere             anywhere 
nixos-fw-accept  all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:ssh
nixos-fw-accept  icmp --  anywhere             anywhere             icmp echo-request
nixos-fw-log-refuse  all  --  anywhere             anywhere

Chain nixos-fw-accept (4 references)
target     prot opt source               destination    
ACCEPT     all  --  anywhere             anywhere       

Chain nixos-fw-log-refuse (1 references)
target     prot opt source               destination    
LOG        tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,ACK/SYN LOG level info prefix "refused connection: "
nixos-fw-refuse  all  --  anywhere             anywhere             PKTTYPE != unicast
nixos-fw-refuse  all  --  anywhere             anywhere 

Chain nixos-fw-refuse (2 references)
target     prot opt source               destination    
DROP       all  --  anywhere             anywhere       

Chain ts-forward (1 references)
target     prot opt source               destination    
MARK       all  --  anywhere             anywhere             MARK xset 0x40000/0xff0000
ACCEPT     all  --  anywhere             anywhere             mark match 0x40000/0xff0000
DROP       all  --  100.64.0.0/10        anywhere       
ACCEPT     all  --  anywhere             anywhere       

Chain ts-input (1 references)
target     prot opt source               destination    
ACCEPT     all  --  portocovo            anywhere       
RETURN     all  --  100.115.92.0/23      anywhere       
DROP       all  --  100.64.0.0/10        anywhere       
ACCEPT     all  --  anywhere             anywhere       
ACCEPT     udp  --  anywhere             anywhere             udp dpt:41641

I have created a simple Ubuntu container:

incus ls
+------+---------+------+----------------------------------------------+-----------+-----------+
| NAME |  STATE  | IPV4 |                     IPV6                     |   TYPE    | SNAPSHOTS |
+------+---------+------+----------------------------------------------+-----------+-----------+
| test | RUNNING |      | fd42:9dc9:1068:2f88:216:3eff:fed9:346 (eth0) | CONTAINER | 0         |
+------+---------+------+----------------------------------------------+-----------+-----------+

It doesn’t get an IPv4 and cannot resolve or connect to the internet:

incus exec test -- ping google.com
ping: google.com: Temporary failure in name resolution
incus exec test -- ping 142.250.184.14
ping: connect: Network is unreachable
incus exec test -- cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .
incus exec test -- resolvectl statusGlobal
       Protocols: -LLMNR -mDNS -DNSOverTLS
                  DNSSEC=no/unsupported
resolv.conf mode: stub

Link 6 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS
                    -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: fe80::216:3eff:fef2:e785%127
       DNS Servers: fe80::216:3eff:fef2:e785%127

From what I read in the forum, it probably has to do with the firewall, but I couldn’t find how to debug or fix more than what I have here. Could you please help? Thanks!

Hi,
you may need to set the following options:

    networking.nftables.enable = true;
    networking.firewall.trustedInterfaces = [ "incusbr*" ];

This bypasses the system firewall rules for incus interfaces (this is not a problem since incus sets it’s own firewall rules)

1 Like

It worked, thanks!