LXD containers can't reach external network in dual ipv4/ipv6 host

LXD
1

LXD version: snap 4.10

Host has both ipv4 and ipv6.

Before adding and using ipv6, everything fine. After start using ipv6, the lxdbri0 networking to the outside word is broken. Containers get internal ipv4 address . Pings from inside containers all work fine. But any real networking is broken as nothing can connect.

Testing various connections finds that lxdbr0 is always attempting the ipv6 route for networking even though lxdbr0 is setup for ipv4 only.

It appears the containers are trying to access external network using ipv6 ip/routes and the lxdbr0 is setup for only ipv4.nat.

EDIT: Both ipv6 and ipv4 pings from inside containers work. However any real tcp port connect fails. Host connects fine.

# lxc network show lxdbr0
config:
  ipv4.address: 10.0.5.1/24
  ipv4.nat: "true"
  ipv6.address: none
description: ""
name: lxdbr0
type: bridge

#ip route
default via 104.1.1.33 dev enp35s0 proto static 
10.0.1.0/24 via 10.0.3.1 dev nas proto static 
10.0.3.0/30 dev nas proto kernel scope link src 10.0.3.1 
10.0.5.0/24 dev lxdbr0 proto kernel scope link src 10.0.5.1 
104.1.1.32/29 dev enp35s0 proto kernel scope link src 104.1.1.34 

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 7406 2146K ACCEPT     all  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */
 2027  777K ACCEPT     all  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */

:POSTROUTING ACCEPT [0:0]
:ts-postrouting - [0:0]
-A POSTROUTING -s 10.0.5.0/24 ! -d 10.0.5.0/24 -m comment --comment "generated for LXD network lxdbr0" -j MASQUERADE
2

Please show output of ip a and ip r inside one of the problem containers.

Also please confirm you have restarted the containers since changing the lxdbr0 to disable IPv6 (so they release their former IPv6 addresses)?

Also, on the LXD host please show output of ip a and ip r and sudo ps aux | grep dnsmasq

3

@tomp

Yes, both container and snap.lxd.daemon was restarted.

Container:

#ip r
default via 10.0.5.1 dev eth0 proto dhcp src 10.0.5.137 metric 100 
10.0.5.0/24 dev eth0 proto kernel scope link src 10.0.5.137 
10.0.5.1 dev eth0 proto dhcp scope link src 10.0.5.137 metric 100 

#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: gre0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
4: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
713: eth0@if714: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:9e:8b:11 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.5.137/24 brd 10.0.5.255 scope global dynamic eth0
       valid_lft 2883sec preferred_lft 2883sec
    inet6 fe80::216:3eff:fe9e:8b11/64 scope link 
       valid_lft forever preferred_lft forever

Host:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp35s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:50:99:d8:de:2f brd ff:ff:ff:ff:ff:ff
    inet 104.1.1.34/29 brd 104.1.1.39 scope global enp35s0
       valid_lft forever preferred_lft forever
    inet6 2607:8888:1:37::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::d250:99ff:fed8:de2f/64 scope link 
       valid_lft forever preferred_lft forever
3: enp36s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:50:99:d8:de:30 brd ff:ff:ff:ff:ff:ff
699: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq state UNKNOWN group default qlen 500
    link/none 
    inet 100.100.190.107/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0:ab12:4843:cd96:6264:be6b/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5f0a:78c0:a3b7:d9ff/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
712: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:bc:04:a1 brd ff:ff:ff:ff:ff:ff
    inet 10.0.5.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:febc:4a1/64 scope link 
       valid_lft forever preferred_lft forever
714: veth84f49de3@if713: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 72:70:95:02:b7:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0
716: vethca5bc0a4@if715: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 52:16:02:83:5b:06 brd ff:ff:ff:ff:ff:ff link-netnsid 1
720: vethfe913f72@if719: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether b2:4d:33:ca:57:11 brd ff:ff:ff:ff:ff:ff link-netnsid 3
722: veth64c47874@if721: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether aa:fa:f9:57:36:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 4
724: veth71edd688@if723: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether c2:f1:4b:08:fe:cf brd ff:ff:ff:ff:ff:ff link-netnsid 5
726: veth6f24888a@if725: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 46:fd:3d:a9:4a:3d brd ff:ff:ff:ff:ff:ff link-netnsid 6
728: vethc8bada18@if727: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 86:4b:5d:80:92:20 brd ff:ff:ff:ff:ff:ff link-netnsid 7
730: veth12812ceb@if729: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether c6:74:72:a1:be:10 brd ff:ff:ff:ff:ff:ff link-netnsid 8
732: vethe29d127d@if731: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether f6:87:43:01:30:5f brd ff:ff:ff:ff:ff:ff link-netnsid 9
734: veth9517adce@if733: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether ae:d1:ae:cd:d8:9b brd ff:ff:ff:ff:ff:ff link-netnsid 10
736: veth8f7dc205@if735: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 52:44:6c:64:15:6c brd ff:ff:ff:ff:ff:ff link-netnsid 11
738: vethf12ba0fa@if737: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether ea:00:82:5e:7d:59 brd ff:ff:ff:ff:ff:ff link-netnsid 12
740: vethfd4e7d30@if739: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 4a:3b:da:d6:a3:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 2
250: gre0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
251: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
252: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
253: nas@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN group default qlen 1000
    link/gre 104.1.1.34 peer 104.2.2.175
    inet 10.0.3.1/30 brd 10.0.3.3 scope global nas
       valid_lft forever preferred_lft forever

#ip r
default via 104.1.1.33 dev enp35s0 proto static 
10.0.1.0/24 via 10.0.3.1 dev nas proto static 
10.0.3.0/30 dev nas proto kernel scope link src 10.0.3.1 
10.0.5.0/24 dev lxdbr0 proto kernel scope link src 10.0.5.1 
104.1.1.32/29 dev enp35s0 proto kernel scope link src 104.1.1.34 

# ps aux | grep dnsmasq
lxd       796199  0.0  0.0  43644  3440 ?        Ss   02:47   0:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.0.5.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.0.5.2,10.0.5.254,1h -s lxd -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd
4

Your container doesn’t have a global IPv6 address, what makes you think it is trying to use IPv6?

5

@tomp

Because when I tried to ping google.com within container, for example, it returned IPV6 address for ping and the ping worked. socket connect failed though.

I tried to ping host that have AAAA records within container and they all defaulted to the ipv6 address.

However, the above ipv6 from ipv4 bridged container problem has now disappeared. Scratching my head right now. Still has the socket connect issue from container but no longer the ipv6 issue. Not sure if several restarts of lxd and container was the fix for the ipv6 issue.

network:
  version: 2
  renderer: networkd
  ethernets:
    enp35s0:
      accept-ra: false

accept-ra: false

The only non-standard netplan config on the ubuntu 20.04 host was adding accept-ra: false so it doesn’t auto assign a temporary ipv6 address.

6

I don’t really understand what you mean by socket connect problem, please can you be more specific and show the commands and resulting output that you are using to test.

I wonder if perhaps dnsmasq is returning AAAA records because your host does have IPv6 connectivity (it appears your external interfaces have IPv6 addresses), but really it shouldn’t because the lxdbr0 interface it listens on doesn’t.

But I would be looking for a clearer reproducer first (because it is possible to query for AAAA records when having no IPv6 connectivity, so I want to rule that out first).

7

@tomp

Just tried apt update within container and it tried to connect via ipv6 address.

root@ci-1:~# apt update
0% [Connecting to archive.ubuntu.com (2001:67c:1360:8001::23)] [Connecting to security.ubuntu.com (2001:67c:1562::15)]

This container is the same as above. Any tcp connection attempt to external network fails. Host has no issue.

When I mean socket connect issue is that ICMP works within container for both IPV4 and IPV6 address. However, any type of TCP SOCKET connect will never able to connect. So telnet port 22/80/443 for example never will never connect. S

8

What OS is the container?

9

Please can you show me that ICMP is working?

10

Please can you show me an ip -6 r from the container too.

11

You shouldn’t need accept-ra: false as there shouldn’t be any router advertisements, please can you remove that and restart your container, so we can see what IPv6 routes are being advertised.

12

I stand corrected. IPV6 icmp doesn’t work.

root@ci-1:~# ping 2607:f8b0:4007:80b::200e
ping: connect: Network is unreachable

#root@ci-1:~# ip -6 r
fe80::/64 dev eth0 proto kernel metric 256 pref medium

Container Ping IPV4 and Telnet to Google.com. ICMP connects, TCP 80 fails.

root@ci-1:~# ping google.com
PING google.com (216.58.193.206) 56(84) bytes of data.
64 bytes from lax02s23-in-f206.1e100.net (216.58.193.206): icmp_seq=1 ttl=120 time=0.656 ms
64 bytes from lax02s23-in-f206.1e100.net (216.58.193.206): icmp_seq=2 ttl=120 time=0.617 ms
64 bytes from lax02s23-in-f206.1e100.net (216.58.193.206): icmp_seq=3 ttl=120 time=0.615 ms
64 bytes from lax02s23-in-f206.1e100.net (216.58.193.206): icmp_seq=4 ttl=120 time=0.611 ms
64 bytes from lax02s23-in-f206.1e100.net (216.58.193.206): icmp_seq=5 ttl=120 time=0.603 ms
64 bytes from lax02s23-in-f206.1e100.net (216.58.193.206): icmp_seq=6 ttl=120 time=0.638 ms
^C
--- google.com ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5087ms
rtt min/avg/max/mdev = 0.603/0.623/0.656/0.018 ms
root@ci-1:~# telnet google.com 80
Trying 216.58.193.206...

Both host and container is Ubuntu 20.04 on 5.10.0-051000-generic mainline kernel.

13

OK so this isn’t anything to do with IPv6.

Looks like it may be a firewall issue on the host.

14

Host itself has none of the above issue. So the only firewall issue is between lxdbr0 and the host?

Unfortunately I have to enable accept-ra: false or else an extra IP and ROUTE is added which breaks my ipv6 out networking for some reason. Maybe related to my network provider and their ipv6 route announcements .

15

Right so there are two issues:

  1. IPv4 connectivity between container and external network is broken. Please check your host’s firewall because just because outbound connectivity from the host to external works, doesn’t mean that your host’s firewall isn’t blocking routed/forwarded traffic (i.e traffic from the container). The output of sudo iptables-save and/or sudo nft list ruleset usually helps diagnose these sorts of issues.

  2. Please can you clarify whether the accept-ra option is on your host or in your container? Because if its in your container, then you shouldn’t need it. If its on your host, then I’m surprised your ISP is advertising routes that break IPv6 connectivity, but if that is the case, then fine to leave disabled, if your IPv6 connectivity is working OK on the host.

16

Are you running docker by any chance? I didn’t notice it in your interface list, but it is well known (on these forums at least) that docker on the host breaks forwarding for LXD containers.

17

No docker in host or container.

  1. I’m double/triple checking the firwall rules right now. nftables is not installed.
  2. Yes, the accept-ra is on Host. Container has dhcp4 in netplan.

Host:

Shows on host both ipv4 and ipv6 works.

#dig AAAA google.com
root@webnx:/etc/netplan# telnet 2607:f8b0:4007:80b::200e 80
Trying 2607:f8b0:4007:80b::200e...
Connected to google.com.
Escape character is '^]'.

#dig A google.com
root@webnx:/etc/netplan# telnet 142.250.68.46 80
Trying 142.250.68.46...
Connected to 142.250.68.46.
Escape character is '^]'.
18

OK good.

So when you open a connection from the host it goes via the iptables OUTPUT chain whereas traffic from the container will go via the FORWARD chain. So different rules can apply.

19

nftables not installed.


Command 'nft' not found, but can be installed with:

apt install nftables

iptables-save

Everything seems good to me. Forwarding enabled for lxdbr0 and lxd injected all the rules like before.

# Generated by iptables-save v1.8.4 on Mon Jan 25 04:45:21 2021
*raw
:PREROUTING ACCEPT [829561:297244864]
:OUTPUT ACCEPT [733936:285769639]
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CT --notrack
-A PREROUTING -p udp -m udp -j CT --notrack
-A OUTPUT -j CT --notrack
COMMIT
# Completed on Mon Jan 25 04:45:21 2021
# Generated by iptables-save v1.8.4 on Mon Jan 25 04:45:21 2021
*mangle
:PREROUTING ACCEPT [557865:196009503]
:INPUT ACCEPT [556276:195878282]
:FORWARD ACCEPT [1601:135157]
:OUTPUT ACCEPT [495556:187949366]
:POSTROUTING ACCEPT [497157:188084523]
-A POSTROUTING -o lxdbr0 -p udp -m udp --dport 68 -m comment --comment "generated for LXD network lxdbr0" -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Jan 25 04:45:21 2021
# Generated by iptables-save v1.8.4 on Mon Jan 25 04:45:21 2021
*nat
:PREROUTING ACCEPT [12630:782487]
:INPUT ACCEPT [12630:782487]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 10.0.5.0/24 ! -d 10.0.5.0/24 -m comment --comment "generated for LXD network lxdbr0" -j MASQUERADE
COMMIT
# Completed on Mon Jan 25 04:45:21 2021
# Generated by iptables-save v1.8.4 on Mon Jan 25 04:45:21 2021
*filter
:INPUT DROP [135:8780]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [495131:187855574]
:SYNFLOOD32 - [0:0]
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 8.8.8.8/32 -p udp -j ACCEPT
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A INPUT -p icmp -m hashlimit --hashlimit-upto 100/sec --hashlimit-burst 200 --hashlimit-mode srcip --hashlimit-name icmp -j ACCEPT
-A INPUT -p udp -m udp --sport 123 -m hashlimit --hashlimit-upto 100/sec --hashlimit-burst 100 --hashlimit-mode srcip --hashlimit-name ntp -j ACCEPT
-A INPUT -p gre -m set --match-set good src -j ACCEPT
-A INPUT -p udp -m set --match-set good src -j ACCEPT
-A INPUT -p tcp -m set --match-set good src -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m set --match-set cloudflare src -j ACCEPT
-A INPUT -p tcp -m tcp -m multiport ! --dports 443 -j DROP
-A INPUT -p tcp -m recent --rcheck --seconds 15 --name BLACKLIST32 --mask 255.255.255.255 --rsource -j DROP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j SYNFLOOD32
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A SYNFLOOD32 -m hashlimit --hashlimit-upto 100/sec --hashlimit-burst 150 --hashlimit-mode srcip --hashlimit-name synflood32 --hashlimit-htable-size 100000 --hashlimit-htable-max 200000 --hashlimit-htable-expire 30000 -j ACCEPT
-A SYNFLOOD32 -m recent --set --name BLACKLIST32 --mask 255.255.255.255 --rsource
-A SYNFLOOD32 -j DROP
COMMIT
# Completed on Mon Jan 25 04:45:21 2021
20

This is likely the problem: